Trying to convert a 7 digit julian date into MMDDYY format - sql

I'm trying to convert a 7 digit julian/mainframe date into a calendar date of format mmddyy in SQL Server.
An example of this being julian date 2005020, where the first 4 digits are the year, 2005, and the last three are the day number in a calendar, so 020 = january 20. So I'd need to get this date as 012005(MMDDYY) in SQL Server.
I've been using the following query but keep getting an error after it loads a few records:
SELECT DATEADD(day,CAST(RIGHT([julianDateColumn],3) as int)-,LEFT([julianDateColumn],4))
The error I've been getting:
Conversion failed when converting date and/or time from character string.
Originally I was doing this in an Access DB using the "DATESERIAL" function but from what I've seen the closest thing to that in SQL Server was "DATEFROMPARTS", I tried using the following formula but it also didn't work:
DATEFROMPARTS([julianDateColumn]/1000,1,[julianDateColumn] % 1000)
Thanks in advance!

The simplest would seem to be to take the left as the year, and the add the days (-1) to make a date. Also, rather than using a format of MMDDYY I'm going to go straight a date datatype. If you want it in a specific format, that's for your presentation layer.
SELECT JulianDate,
CONVERT(date,DATEADD(DAY,RIGHT(JulianDate,3)-1,CONVERT(datetime,LEFT(JulianDate,4)))) AS ActualDate --4 int strings are iterpreted as the year, so I'm going to take advantage of that
FROM (VALUES('2005020'))V(JulianDate);
Based on the comments on the answer, it appears that the OP has some dates that don't conform to the format that stated (yyyyddd). Therefore what we could use here is a calendar table, here, and then LEFT JOIN to it and see what bad rows you get (and INNER JOIN to get the dates).
You can create the table with something like this:
CREATE TABLE dbo.CalendarTable (CalendarDate date NOT NULL PRIMARY KEY,
CalenderYear AS DATEPART(YEAR, CalendarDate) PERSISTED,
CalenderMonth AS DATEPART(MONTH, CalendarDate) PERSISTED,
CalenderDay AS DATEPART(DAY, CalendarDate) PERSISTED,
CalenderMonthName AS DATENAME(MONTH, CalendarDate),
JulianDate AS DATEPART(YEAR,CalendarDate) * 1000 + DATEDIFF(DAY,DATEFROMPARTS(DATEPART(YEAR, CalendarDate),1,1),CalendarDate) + 1 PERSISTED); --Some example columns
WITH N AS(
SELECT N
FROM (VALUES(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL))N(N)),
Tally AS(
SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) -1 AS I
FROM N N1, N N2, N N3, N N4, N N5, N N6),
Dates AS(
SELECT CONVERT(date,DATEADD(DAY, T.I, '19000101')) AS CalendarDate
FROM Tally T)
INSERT INTO dbo.CalendarTable(CalendarDate)
SELECT CalendarDate
FROM Dates
WHERE CalendarDate < '21000101';
GO
Then we can do something like this to get the bad rows:
SELECT YT.JulianDate
FROM dbo.YourTable YT
LEFT JOIN dbo.CalendarTable CT ON YT.JulianDate = CT.JulianDate
WHERE CT.JulianDate IS NULL;

I think I would use datefromparts() and dateadd():
select dateadd(day,
right(juliandate, 3) - 1),
datefromparts(left(juliandate, 4), 1, 1)
)

select Format(cast(concat(substring('2005020', 1, 4), '-', Month(cast(substring('2005020', 5, len('2005020')) as int)) ,'-', day(dateadd(day,-1,cast(substring('2005020', 5, len('2005020')) as int ) ) )) as date), 'MMddyy')

Related

Get data from the past "x" hours excluding the weekend

I'm using Oracle 11g.
What I would like to do is select login data from the past (let's say) 10 hours. I would like to exclude the weekend from this, and let my query wrap around to the previous Friday. For example, if it's Monday at 8:00 AM, I want to be able to find the logins that occurred from 12-8 AM Monday, as well as data from 10-11:59 PM on Friday.
My current statement looks like this:
select * from logins where end_time >= sysdate - (10) / (24)
But I'm not sure how to exclude Saturday and Sunday.
I didn't see what Database as the Question was only tagged as SQL. Regardless the answer below is SQL Server but the concept will work in any database. I've used the method shown below in SQLite and Oracle.
Get a list of dates between two dates using a function
There is an excellent answer contained in the above SO Post I provided above. However this question has so many answers and the excepted answer is actually not the answer that solved my issue when I originally had a similar calendar table problem on my end.
I've pasted below the answer that works without having to create a calendar table and works within any standard query which can then be turned into a View/Stored Proc/Function etc.... And since the SQL below does not use a recursive CTE you can use this code in Views, Stored Procedures, Functions... etc... The dates table is still populated using a CTE but it does this using an exponential method of joining in several fake tables. I think it is brilliant and it performs well.
DECLARE #StartDate DATE = '08/01/2021'
, #EndDate Date = '08/01/2022'
;
-- From http://stackoverflow.com/questions/1378593/get-a-list-of-dates-between-two-dates-using-a-function
-- it basically just selects a whole bunch of 1s, gets the ROW_NUMBER() for each and filters.
-- This seems to perform better than inserting a bunch of rows in a WHILE loop or a recursive CTE
-- just UNIONing to the next date.
WITH
N0 AS (SELECT 1 AS n UNION ALL SELECT 1)
,N1 AS (SELECT 1 AS n FROM N0 t1, N0 t2)
,N2 AS (SELECT 1 AS n FROM N1 t1, N1 t2)
,N3 AS (SELECT 1 AS n FROM N2 t1, N2 t2)
,N4 AS (SELECT 1 AS n FROM N3 t1, N3 t2)
,N5 AS (SELECT 1 AS n FROM N4 t1, N4 t2)
,N6 AS (SELECT 1 AS n FROM N5 t1, N5 t2)
,nums AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS num FROM N6)
SELECT DATEADD(day, num - 1, #StartDate) AS TheDate
, DATENAME(WEEKDAY, (DATEADD(day, num - 1, #StartDate))) AS TheDayName
, num - 1 AS TheOffset
FROM nums
WHERE num <= DATEDIFF(day, #StartDate, #EndDate) + 1
In the above SQL I added the field for TheDayName so that you can filter out the weekends however you can add so many more fields that pull in more details of the dates being pulled in your query. Refer to the below SQL as it shows how to add all of the unique date parts into the CTE as needed. Again I added one of the below fields to the above CTE to pull TheDayName.
/* The below shows how to obtain additional information a date */
-- https://www.mssqltips.com/sqlservertip/4054/creating-a-date-dimension-or-calendar-table-in-sql-server/
DECLARE #d DATE = '07/26/21';
SELECT
TheDate = CONVERT(date, #d),
TheDay = DATEPART(DAY, #d),
TheDayName = DATENAME(WEEKDAY, #d),
TheWeek = DATEPART(WEEK, #d),
TheISOWeek = DATEPART(ISO_WEEK, #d),
TheDayOfWeek = DATEPART(WEEKDAY, #d),
TheMonth = DATEPART(MONTH, #d),
TheMonthName = DATENAME(MONTH, #d),
TheQuarter = DATEPART(Quarter, #d),
TheYear = DATEPART(YEAR, #d),
TheFirstOfMonth = DATEFROMPARTS(YEAR( #d), MONTH( #d ), 1),
TheLastOfYear = DATEFROMPARTS(YEAR( #d), 12, 31),
TheDayOfYear = DATEPART(DAYOFYEAR, #d)
Last if you are needing an example of how you JOIN back to a calendar table see the accepted answer in this SO Post: How to Calculate the Total Unique Days Employed for All Jobs - No overlap days counted twice

How to get six weeks data from a week column?

I have a legacy query in which I am looking data for six weeks as shown below. In my below AND condition I get data for past six weeks and it worked fine in 2020 middle and end. But since 2021 started, this stopped working because of obvious subtraction I am doing with 6.
AND data.week_col::integer BETWEEN DATE_PART(w, CURRENT_DATE) - 6 AND DATE_PART(w, CURRENT_DATE) - 1
There is a bug in above query because of which it stopped working in 2021. How can I change above condition so that it can work entire year without any issues and give me data for past 6 weeks.
Update
Below is my query which I am running:
select *,
dateadd(d, - datepart(dow, trunc(CONVERT_TIMEZONE('UTC','PST8PDT',client_date))), trunc(CONVERT_TIMEZONE('UTC','PST8PDT',client_date)) + 6) as day,
date_part(week, day) as week_col
from holder data
where data.week_col::integer BETWEEN DATE_PART(w, CURRENT_DATE) - 6 AND DATE_PART(w, CURRENT_DATE) - 1
client_date column has values like this - 2021-01-15 21:30:00.0. And from that I get value of day column and from day column I get value of
week_col column as shown above.
week_col column has values like 53, 52 .... It's a week number in general.
Because of my AND condition I am getting data for week 1 only but technically I want data for 49, 50, 51, 52, 53 and 1 as it is past six weeks. Can I use day column here to get correct past six weeks?
Would this serve as a solution? I do not know much about the redshirt syntax but I read it supports dateadd(). If you are normalizing client_date to a time zone converted day with no time then why not simply use that in the comparison to the current date converted to the same time zone.
WHERE
client_date BETWEEN
DATEADD(WEEK,-6,trunc(CONVERT_TIMEZONE('UTC','PST8PDT',CURRENT_DATE)))
AND
DATEADD(WEEK,-1,trunc(CONVERT_TIMEZONE('UTC','PST8PDT',CURRENT_DATE)))
If the above logic works out then you may want to convert the -6 and -1 week to variables, if that is supported.
Solution 2
This is a bit more verbose but involves virtualizing a calender table and then joining your current date parameter into the calender data, for markers. Finally, you can join your data against the calender which has been normalized by weeks in time chronologically.
This is SQL Server syntax, however, I am certain it can be converted to RS.
DECLARE #D TABLE(client_date DATETIME)
INSERT #D VALUES
('11/20/2020'),('11/27/2020'),
('12/4/2020'),('12/11/2020'),('12/18/2020'),('12/25/2020'),
('01/8/2021'),('01/8/2021'),('1/15/2021'),('1/22/2021'),('1/29/2021')
DECLARE #Date DATETIME = '1/23/2021'
DECLARE #StartDate DATETIME = '01/01/2010'
DECLARE #NumberOfDays INT = 6000
;WITH R1(N) AS (SELECT 1 FROM (VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1))dt(n)),
R2(N) AS (SELECT 1 FROM R1 a, R1 b),
R3(N) AS (SELECT 1 FROM R2 a, R2 b),
Tally(Number) AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) FROM R3)
,WithTally AS
(
SELECT CalendarDate = DATEADD(DAY,T.Number,#StartDate)
FROM Tally T
WHERE T.Number < #NumberOfDays
)
,Calendar AS
(
SELECT
CalendarDate,
WeekIndex = DENSE_RANK() OVER(ORDER BY DATEPART(YEAR, CalendarDate), DATEPART(WEEK, CalendarDate))
FROM
WithTally
),
CalendarAlignedWithCurrentDateParamater AS
(
SELECT *
FROM
Calendar
CROSS JOIN (SELECT WeekIndexForToday=WeekIndex FROM Calendar WHERE Calendar.CalendarDate=#Date ) AS X
)
SELECT
D.*,
C.WeekIndex,
C.WeekIndexForToday
FROM
CalendarAlignedWithCurrentDateParamater C
INNER JOIN #D D ON D.client_date = C.CalendarDate
WHERE
C.WeekIndex BETWEEN C.WeekIndexForToday-6 AND C.WeekIndexForToday-1
OPTION (MAXRECURSION 0)

SQL: "Cannot construct data type date" when comparing two dates

I'm having an issue in a query where SQL Server is throwing the error
Cannot construct data type date, some of the arguments have values which are not valid
when comparing two date objects that themselves are valid.
If I remove the where clause, it resolves without error, but the moment I try to compare them with any relational or equality operator it errors again.
Minimum query to reproduce the issue is as follows:
with Years as
(
select
YEAR(getdate()) + 1 Year,
DATEFROMPARTS(YEAR(getdate()) + 1, 1, 1) FirstOfTheYear,
0 YearOffset
union all
select
Year - 1,
DATEFROMPARTS(Year - 1, 1, 1),
YearOffset + 1
from Years
where YearOffset < 5
),
Months as
(
select 1 Month
union all
select Month + 1
from Months
where Month < 12
),
Days as
(
select 1 Day
union all
select Day + 1
from Days
where Day < 31
),
Dates as
(
select cast(DATEFROMPARTS(Year, Month, Day) as date) Date
from Years
cross join Months
cross join Days
where DAY(EOMONTH(FirstOfTheYear, Month - 1)) >= Day
)
select Dates.Date, cast ('2019-10-01' as date), CAST ('2019-10-11' as date)
from Dates
where Date = cast ('2019-10-01' as date) -- Comment this line out and the error goes away, occurs with any date construction pattern
--where Dates.[Date] >= datefromparts(2019, 10, 01) and Dates.[Date] <= DATEFROMPARTS(2019, 10, 11)
order by date
Commenting out the where clause returns results as expected, confirming that it is specifically the comparison that is triggering this issue.
Additionally, manually creating a handful of dates (first of the year, 2015-2019, the October dates in the query) and querying against that does not cause the error to show.
Edit: I want to emphasize that the code is already handling February and leap years correctly. The output of the Dates CTE is valid and outputs the full range without error. It is only when I reference the date in the where clause that it throws the error
Edit2: I was able to resolve my issue by switching to a different date generation pattern (adding a day, day by day, in a recursive), but I still am curious what causes this error.
The point of a couple of the other answers is that attacking the issue in the manner you are is not necessarily the most efficient way of generating a date's table. Most of the time when constrained with SQL server people will lead someone to use a Tally table for this purpose. Doing so will remain a SET based operation rather than requiring looping or recursion. Which means the recursion limit you mentioned in one of your comments simply doesn't apply.
A Tally table is a set of numeric values that you can then use to generate or produce the values you want. In this case that is approximately 1827 days (5 years + 1 day) but can differ by leap years. The leap years and February are likely the issues within your code. Anyway to generate a tally table you can start with 10 values then cross join till you get to an acceptable number of combinations. 3 cross joins will bring you to 10,000 values and ROW_NUMBER() - 1 can be used to generate a 0 based increment. After which you can use DATEADD() to actually create the dates:
;WITH cteTen AS (
SELECT n FROM (VALUES (0),(0),(0),(0),(0),(0),(0),(0),(0),(0)) T(n)
)
, cteTally AS (
SELECT
N = ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) - 1
FROM
cteTen t10
CROSS JOIN cteTen t100
CROSS JOIN cteTen t1000
CROSS JOIN cteTen t10000
)
, cteStartOfNextYear AS (
SELECT
StartOfNextYear = s.[Date]
,NumOfDaysBetween = DATEDIFF(DAY,DATEADD(YEAR,-5,s.[Date]),s.[Date])
FROM
(VALUES (DATEFROMPARTS(YEAR(GETDATE()) + 1, 1, 1))) s([Date])
)
, cteDates AS (
SELECT
[Date] = DATEADD(DAY,- t.N, s.StartOfNextYear)
FROM
cteStartOfNextYear s
INNER JOIN cteTally t
ON t.N <= NumOfDaysBetween
)
SELECT *
FROM
cteDates
ORDER BY
[Date]
Per our conversation, I see why you would think that EOMONTH() would take care of the issue but it is an order of operations sort of. So the DATEFROMPARTS() portion is analyzed across the entirety of the dataset prior to interpreting the where clause. So it is trying to build the date of 29,30 of Feb. etc. before it is limiting it to the number of days defined by EOMONTH() where clause
I have no idea why you are using code like that to generate days. Why not start at the first date and just add one date at a time?
In any case Feb 29 or 30 or 31 is going to cause an error. You can fix this approach by changing the dates subquery:
Dates as (
select try_convert(date, concat(year, '-' month, '-', day)) as Date
from Years y cross join
Months m cross join
Days
where try_convert(date, concat(year, '-' month, '-', day)) and
DAY(EOMONTH(FirstOfTheYear, Month - 1)) >= Day
)
You're asking DATEFROMPARTS to convert invalid combinations of dates and times. That's what is throwing the error - not your CAST statement.
See Using T-SQL DATEFROMPARTS to return NULL instead of throw error to find your problem dates in general.
Your query creates dates including February 29th, 30th and 31st, as well as the 31st of April, June, September and November.
If you just want all the dates from 2015 through 2020, you can count off a bunch of days and add to a base date. SQL Server will handle the month issues for you:
-- Create up to 16 million integers
WITH N AS (SELECT 0 AS N FROM (VALUES (0),(1),(2),(3),(4),(5),(6),(7)) T(n))
, M AS (SELECT 0 AS N FROM N A, N B, N C, N D, N E, N F, N G, N H)
, Z AS (SELECT ROW_NUMBER() OVER (ORDER BY A.N) AS N FROM M A)
-- Filter only the integers you need; add to a start date
SELECT CAST(DATEADD(DAY, N-1, '2015-01-01') AS DATE) FROM Z
WHERE N < DATEDIFF(DAY, '2015-01-01', '2020-01-01')

Query to check number of records created in a month.

My table creates a new record with timestamp daily when an integration is successful. I am trying to create a query that would check (preferably automated) the number of days in a month vs number of records in the table within a time frame.
For example, January has 31 days, so i would like to know how many days in january my process was not successful. If the number of records is less than 31, than i know the job failed 31 - x times.
I tried the following but was not getting very far:
SELECT COUNT (DISTINCT CompleteDate)
FROM table
WHERE CompleteDate BETWEEN '01/01/2015' AND '01/31/2015'
Every 7 days the system executes the job twice, so i get two records on the same day, but i am trying to determine the number of days that nothing happened (failures), so i assume some truncation of the date field is needed?!
One way to do this is to use a calendar/date table as the main source of dates in the range and left join with that and count the number of null values.
In absence of a proper date table you can generate a range of dates using a number sequence like the one found in the master..spt_values table:
select count(*) failed
from (
select dateadd(day, number, '2015-01-01') date
from master..spt_values where type='P' and number < 365
) a
left join your_table b on a.date = b.CompleteDate
where b.CompleteDate is null
and a.date BETWEEN '01/01/2015' AND '01/31/2015'
Sample SQL Fiddle (with count grouped by month)
Assuming you have an Integers table*. This query will pull all dates where no record is found in the target table:
declare #StartDate datetime = '01/01/2013',
#EndDate datetime = '12/31/2013'
;with d as (
select *, date = dateadd(d, i - 1 , #StartDate)
from dbo.Integers
where i <= datediff(d, #StartDate, #EndDate) + 1
)
select d.date
from d
where not exists (
select 1 from <target> t
where DATEADD(dd, DATEDIFF(dd, 0, t.<timestamp>), 0) = DATEADD(dd, DATEDIFF(dd, 0, d.date), 0)
)
Between is not safe here
SELECT 31 - count(distinct(convert(date, CompleteDate)))
FROM table
WHERE CompleteDate >= '01/01/2015' AND CompleteDate < '02/01/2015'
You can use the following query:
SELECT DATEDIFF(day, t.d, dateadd(month, 1, t.d)) - COUNT(DISTINCT CompleteDate)
FROM mytable
CROSS APPLY (SELECT CAST(YEAR(CompleteDate) AS VARCHAR(4)) +
RIGHT('0' + CAST(MONTH(CompleteDate) AS VARCHAR(2)), 2) +
'01') t(d)
GROUP BY t.d
SQL Fiddle Demo
Explanation:
The value CROSS APPLY-ied, i.e. t.d, is the ANSI string of the first day of the month of CompleteDate, e.g. '20150101' for 12/01/2015, or 18/01/2015.
DATEDIFF uses the above mentioned value, i.e. t.d, in order to calculate the number of days of the month that CompleteDate belongs to.
GROUP BY essentially groups by (Year, Month), hence COUNT(DISTINCT CompleteDate) returns the number of distinct records per month.
The values returned by the query are the differences of [2] - 1, i.e. the number of failures per month, for each (Year, Month) of your initial data.
If you want to query a specific Year, Month then just simply add a WHERE clause to the above:
WHERE YEAR(CompleteDate) = 2015 AND MONTH(CompleteDate) = 1

Can I use recursion in a Sql Server 2005 View?

I tried to use OPTION (MAXRECURSION 0) in a view to generate a list of dates.
This seems to be unsupported. Is there a workaround for this issue?
EDIT to Explain what I actually want to do:
I have 2 tables.
table1: int weekday, bool available
table2: datetime date, bool available
I want the result:
view1: date (here all days in this year), available(from table2 or from table1 when not in table2).
That means I have to apply a join on a date with a weekday.
I hope this explanation is understandable, because I actually use more tables with more fields in the query.
I found this code to generate the recursion:
WITH Dates AS
(
SELECT cast('2008-01-01' as datetime) Date
UNION ALL
SELECT Date + 1
FROM Dates
WHERE Date + 1 < DATEADD(yy, 1, GETDATE())
)
No - if you can find a way to do it within 100 levels of recusion (have a table of numbers), which will get you to within 100 recursion levels, you'll be able to do it. But if you have a numbers or pivot table, you won't need the recursion anyway...
See this question (but I would create a table and not a table-valued function), this question and this link and this link
You can use a CTE for hierarchical queries.
Here you go:
;WITH CTE_Stack(IsPartOfRecursion, Depth, MyDate) AS
(
SELECT
0 AS IsPartOfRecursion
,0 AS Dept
,DATEADD(DAY, -1, CAST('01.01.2012' as datetime)) AS MyDate
UNION ALL
SELECT
1 AS IsPartOfRecursion
,Parent.Depth + 1 AS Depth
--,DATEADD(DAY, 1, Parent.MyDate) AS MyDate
,DATEADD(DAY, 1, Parent.MyDate) AS MyDate
FROM
(
SELECT 0 AS Nothing
) AS TranquillizeSyntaxCheckBecauseWeDontHaveAtable
INNER JOIN CTE_Stack AS Parent
--ON Parent.Depth < 2005
ON DATEADD(DAY, 1, Parent.MyDate) < DATEADD(YEAR, 1, CAST('01.01.2012' as datetime))
)
SELECT * FROM CTE_Stack
WHERE IsPartOfRecursion = 1
OPTION (MAXRECURSION 367) -- Accounting for leap-years
;