I've been searching for quite a while and can't find an answer to this question:
I have a PostgreSQL table that is staged the following way:
Start Date | End Date | Name | Team
-----------+------------+---------+------
2017-10-01 | 2017-10-10 | Person | 1
And what I would like to is have each row a day between the start date and end date with the corresponding name and team of the person:
Date | Name | Team
------------+---------+-------
2017-10-01 | Person | 1
------------+---------+-------
2017-10-02 | Person | 1
------------+---------+-------
2017-10-03 | Person | 1
Is it even possible to do this with PostgreSQL? I'm currently running PostgreSQL 9.3.
You can use generate_series() for that:
select t.dt::date, p.name, p.team
from person p, generate_series(p.start_date, p.end_date, interval '1' day) as t(dt)
order by t.dt::date;
I don't have 9.3 around any more, but I think that should also work with that old version.
Related
Given a simple data model that consists of a user table and a check_in table with a date field, I want to calculate the retention date of my users. So for example, for all users with one or more check ins, I want the percentage of users who did a check in on their 2nd day, on their 3rd day and so on.
My SQL skills are pretty basic as it's not a tool that I use that often in my day-to-day work, and I know that this is beyond the types of queries I am used to. I've been looking into pivot tables to achieve this but I am unsure if this is the correct path.
Edit:
The user table does not have a registration date. One can assume it only contains the ID for this example.
Here is some sample data for the check_in table:
| user_id | date |
=====================================
| 1 | 2020-09-02 13:00:00 |
-------------------------------------
| 4 | 2020-09-04 12:00:00 |
-------------------------------------
| 1 | 2020-09-04 13:00:00 |
-------------------------------------
| 4 | 2020-09-04 11:00:00 |
-------------------------------------
| ... |
-------------------------------------
And the expected output of the query would be something like this:
| day_0 | day_1 | day_2 | day_3 |
=================================
| 70% | 67 % | 44% | 32% |
---------------------------------
Please note that I've used random numbers for this output just to illustrate the format.
Oh, I see. Assuming you mean days between checkins for users -- and users might have none -- then just use aggregation and window functions:
select sum( (ci.date = ci.min_date)::numeric ) / u.num_users as day_0,
sum( (ci.date = ci.min_date + interval '1 day')::numeric ) / u.num_users as day_1,
sum( (ci.date = ci.min_date + interval '2 day')::numeric ) / u.num_users as day_2
from (select u.*, count(*) over () as num_users
from users u
) u left join
(select ci.user_id, ci.date::date as date,
min(min(date::date)) over (partition by user_id order by date) as min_date
from checkins ci
group by user_id, ci.date::date
) ci;
Note that this aggregates the checkins table by user id and date. This ensures that there is only one row per date.
My goal is to join a sales program table to a calendar table so that there would be a joined table with the full trailing 52 weeks by day, and then the sales data would be joined to it. The idea would be that there are nulls I could COALESCE after the fact. However, my problem is that I only get results without nulls from my sales data table.
The questions I've consulted so far are:
Join to Calendar Table - 5 Business Days
Joining missing dates from calendar table Which points to
MySQL how to fill missing dates in range?
My Calendar table is all 364 days previous to today (today being day 0). And the sales data has a program field, a store field, and then a start date and an end date for the program.
Here's what I have coded:
SELECT
CAL.DATE,
CAL.DAY,
SALES.ITEM,
SALES.PROGRAM,
SALES.SALE_DT,
SALES.EFF_BGN_DT,
SALES.EFF_END_DT
FROM
CALENDAR_TABLE AS CAL
LEFT JOIN
SALES_TABLE AS SALES
ON CAL.DATE = SALES.SALE_DT
WHERE 1=1
and SALES.ITEM = 1 or SALES.ITEM is null
ORDER BY DATE ASC
What I expected was 365 records with dates where there were nulls and dates where there were filled in records. My query resulted in a few dates with null values but otherwise just the dates where a program exists.
DATE | ITEM | PROGRAM | SALE_DT | PRGM_BGN | PRGM_END |
----------|--------|---------|----------|-----------|-----------|
8/27/2020 | | | | | |
8/26/2020 | | | | | |
8/25/2020 | | | | | |
8/24/2020 | | | | | |
6/7/2020 | 1 | 5 | 6/7/2020 | 2/13/2016 | 6/7/2020 |
6/6/2020 | 1 | 5 | 6/6/2020 | 2/13/2016 | 6/7/2020 |
6/5/2020 | 1 | 5 | 6/5/2020 | 2/13/2016 | 6/7/2020 |
6/4/2020 | 1 | 5 | 6/4/2020 | 2/13/2016 | 6/7/2020 |
Date = Calendar day.
Item = Item number being sold.
Program = Unique numeric ID of program.
Sale_Dt = Field populated if at least one item was sold under this program.
Prgm_bgn = First day when item was eligible to be sold under this program.
Prgm_end = Last day when item was eligible to be sold under this program.
What I would have expected would have been records between June 7 and August 24 which just had the DATE column populated for each day and null values as what happens in the most recent four records.
I'm trying to understand why a calendar table and what I've written are not providing the in-between dates.
EDIT: I've removed the request for feedback to shorten the question as well as an example I don't think added value. But please continue to give feedback as you see necessary.
I'd be more than happy to delete this whole question or have someone else give a better answer, but after staring at the logic in some of the answers in this thread (MySQL how to fill missing dates in range?) long enough, I came up with this:
SELECT
CAL.DATE,
t.* EXCEPT (DATE)
FROM
CALENDER_TABLE AS CAL
LEFT JOIN
(SELECT
CAL.DATE,
CAL.DAY,
SALES.ITEM,
SALES.PROGRAM,
SALES.SALE_DT,
SALES.EFF_BGN_DT,
SALES.EFF_END_DT
FROM
CALENDAR_TABLE AS CAL
LEFT JOIN
SALES_TABLE AS SALES
ON CAL.DATE = SALES.SALE_DT
WHERE 1=1
and SALES.ITEM = 1 or SALES.ITEM is null
ORDER BY DATE ASC) **t**
ON CAL.DATE = t.DATE
From what I can tell, it seems to be what I needed. It allows for the subquery to connect a date to all those records, then just joins on the calendar table again solely on date to allow for those nulls to be created.
Below I have the tables and query which output the below
Table1
EmployeeID | StartDateTimestamp | CohortID | CohortName
---------- | ------------------ | -------- | ----------
1 | 20080101 01:30:00 | 1 | Peanut
1 | 20090204 01:01:00 | 2 | Apple
2 | 20190107 05:52:14 | 1 | Peanut
3 | 20190311 02:35:26 | 2 | Apple
Employee
EmployeeID | HireStartName | StartDateTimestamp2
---------- | ------------- | -------------------
1 | HiredStart | 20080501 01:30:00
1 | DeferredStart | 20090604 01:01:00
2 | HiredStart | 20190115 05:52:14
3 | HiredStart | 20190330 02:35:26
Query
select
t.cohortid,
min(e.startdatetimestamp2) first,
max(e.startdatetimestamp2) last
from table1 t
inner join employee e on e.employeeid = t.employeeid
group by t.cohort_id
Output
ID | FIRST | LAST
1 |20190106 12:00:05 |20180214 03:45:12
2 |20180230 01:45:23 |20180315 01:45:23
My attempt:
SELECT DATE_DIFF(first, last, Day), ID, max(datecolumn1) first, min(datecolumn1) last
Error: Unrecognized name.
How do I enter the reference alias first and last in a Date_Diff?
Do I need to derive a table?
Clarity: Trying to avoid inputting in the dates, since I am looking to find the date diff of both first and last columns for as many rows as there is data.
This answer has been discussed here: Date Difference between consecutive rows
DateDiff has deprecated, and now it is Date_Diff (first, last, day)
Then I tried:
SELECT ID, DATE_DIFF(PARSE_DATE('%y%m%d',t.first), PARSE_DATE('%y%m%d',t.last), DAY) days
FROM table
Failed to parse input string "20180125 01:00:05"
Tried this
SELECT CohortID, date_diff(first,last,day) as days
FROM (select cohortid,min(startdatetimestamp2) first,
max(startdatetimestamp2) last
FROM employee
JOIN table1 on table1.employeeid = employee.employeeid
group by cohortid)
I get days not found on either side of join
Regarding your first question about using aliases in a query, there are some restriction where to use them, specially in the FROM, GROUP BY and ORDER BY statements. I encourage you to have a look here to check these restrictions.
About your main issue, obtaining the date difference between two dates. I would like to point that your timestamp data, in both of your tables, are actually considered as DATETIME format in BigQuery. Therefore, you should use DATETIME builtin functions to get the desired results.
The below query uses the data you provided to obtain the aimed output.
WITH
data AS
(
SELECT
t.cohortid AS ID,
PARSE_DATETIME('%Y%m%d %H:%M:%S', MIN(e.startdatetimestamp2)) AS first,
PARSE_DATETIME('%Y%m%d %H:%M:%S', MAX(e.startdatetimestamp2)) AS last
FROM
`test-proj-261014.sample.table1` t
INNER JOIN
`test-proj-261014.sample.employee` e
ON
e.employeeid = t.employeeid
GROUP BY t.cohortid
)
SELECT
ID,
first,
last,
DATETIME_DIFF(last, first, DAY ) AS diff_days
FROM
data
And the output:
Notice that I created a temp table to format the fields StartDateTimestamp and StartDateTimestamp2, using the PARSE_DATETIME(). Afterwards, I used the DATETIME_DIFF() method to obtain the difference in days between the two fields.
+----------------------------+ +----------------------------+
| ID | Date | | ID | Date |
+----------------------------+ +----------------------------+
| 1 | 2017-06-13 22:10:01 | | 1 | 2017-06-20 22:10:50 |
+----------------------------+ +----------------------------+
| 2 | 2017-06-14 13:22:20 | | 2 | 2017-06-23 22:10:55 |
+----------------------------+ +----------------------------+
| 3 | 2017-06-16 22:10:01 | | 3 | 2017-06-27 22:10:35 |
+----------------------------+ +----------------------------+
| 4 | 2017-07-04 22:10:07 |
+----------------------------+
I have 2 tables, same columns. The first one will be a sample, having entries on a week basis, each with 1-minute difference (10 080 entries in total). While the other one gets new values everytime (1 min), indefinitely.
What I try to do is compare each new entry with an entry from the first database. But I want to compare depending on weekday and time (same hour and same minutes).
For instance, an entry on Monday at 11:00 (whatever month, day and year) should be compared with one on Monday at 11:00 from the sample database.
What I want to do is get the number of the entry from sample database when it's the same weekday as the entry in the second table:
2017-06-20 22:10:50 should return 1.
2017-06-23 22:10:55 should return 3.
2017-06-27 22:10:35 should return 1.
2017-07-04 22:10:07 should return 1.
Edit2:
I think you may understand things better when I explain the purpose of the two tables.
The real tables in my database have more columns: sensors' id and their value.
The first table, sample table, will have data received for a whole week. It's used as a reference.
The second table receives data every minute, when one gets in it should be compared with a record in the sample table to detect if values are equals or not(anormal value).
So I want to detect abnormal values by comparing with a record of same weekday, same hours and minutes.
Well I think I found what I was looking for. This is a useful link : Get month from DATETIME in sqlite
Here is the way to match timestamps :
//Matching weekday
select * from table where strftime('%w', Date) = '0';
//Matching weekday + hours and minutes
select * from table where strftime('%w', Date) = '0' and strftime('%H:%M', Date) = '22:10';
Thank you guys anyway.
In my code using SQL Server, I am comparing data between two months where I have the exact dates identified. I am trying to find if the value in a certain column changes in a bunch of different scenarios. That part works, but what I'd like to do is make it so that I don't have to always go back to change the date each time I wanted to get the results I'm looking for. Is this possible?
My thought was that adding a WITH clause, but it is giving me an aggregation error. Is there anyway I can go about making this date problem simpler? Thanks in advance
EDIT
Ok I'd like to clarify. In my WITH statement, I have:
select distinct
d.Date
from Database d
Which returns:
+------+-------------+
| | Date |
+------+-------------|
| 1 | 01-06-2017 |
| 2 | 01-13-2017 |
| 3 | 01-20-2017 |
| 4 | 01-27-2017 |
| 5 | 02-03-2017 |
| 6 | 02-10-2017 |
| 7 | 02-17-2017 |
| 8 | 02-24-2017 |
| 9 | ........ |
+------+-------------+
If I select this statement and execute, it will return just the dates from my table as shown above. What I'd like to do is be able to have sql that will pull from these date values and compare the last date value from one month to the last date value of the next month. In essence, it should compare the values from date 8 to values from date 4, but it should be dynamic enough that it can do the same for any two dates without much tinkering.
If I didn't misunderstand your request, it seems you need a numbers table, also known as a tally table, or in this case a calendar table.
Recommended post: https://dba.stackexchange.com/questions/11506/why-are-numbers-tables-invaluable
Basically, you create a table and populate it with numbers of year's week o start and end dates. Then join your main query to this table.
+------+-----------+----------+
| week | startDate | endDate |
+------+-----------+----------+
| 1 | 20170101 | 20170107 |
| 2 | 20170108 | 20170114 |
+------+-----------+----------+
Select b.week, max(a.data) from yourTable a
inner join calendarTable b
on a.Date between b.startDate and b.endDate
group by b.week
dynamic dates to filter by BETWEEN
select dateadd(m,-1,dateadd(day,-(datepart(day,cast(getdate() as date))-1),cast(getdate() as date))) -- 1st date of last month
select dateadd(day,-datepart(day,cast(getdate() as date)),cast(getdate() as date)) -- last date of last month
select dateadd(day,-(datepart(day,cast(getdate() as date))-1),cast(getdate() as date)) -- 1st date of current month
select dateadd(day,-datepart(day,dateadd(m,1,cast(getdate() as date))),dateadd(m,1,cast(getdate() as date))) -- last date of the month