Dears,
We have a database that creates a new table for each new day. The naming of the tables follows this pattern: History_tbl_[year]_[month]_[day]. A sample name of the tables for the last 5 days for example is:
History_tbl_2021_10_02
History_tbl_2021_10_01
History_tbl_2021_09_30
History_tbl_2021_09_29
History_tbl_2021_09_28
My goal is to be able to query all the tables from a given date range at once. I can manually select the tables with union all, but it takes lots of time especially if I want to do a long date range. Is there a better way to solve this?
Note: Unfortunately, I don't have the privilege to change the structure and make all data being stored in a single table.
One bare-bones method to generate a list of unioned queries and dynamically execute them would be as follows, tweak as necessary:
declare #from date='20211001', #to date='20211004', #sql nvarchar(max)='';
with d as (
select DateAdd(day, number, #from) dt
from master..spt_values
where type = 'P'
and DateAdd(day, number, #from) <= #to
)
select #sql=String_Agg(qry,' ')
from (
select 'select col1, col2, col3 from History_tbl_'
+ Concat(Year(dt),'_',Right(Concat('0',month(dt)),2),'_', Right(Concat('0',Day(dt)),2))
+ Iif( dt=Max(dt) over() ,'', ' union all') qry
from d
)x
exec sp_executesql #sql
Note that the CTE generates the date range on the fly, ideally you would use a permanent calendar table
Related
I have 1000 tables in my SQL Server database. Table names like CIP_01_12_2022_10_15_20 that I have 1000 table. Table name saved as data time(CIP_DD_MM_YYYY_HH_MM_SS).
So I need to delete like between particular dates.
Tables named like below
CIP_01_12_2022_10_15_20
CIP_01_12_2022_10_15_25
CIP_01_12_2022_10_15_35
CIP_01_12_2022_10_15_45
CIP_01_12_2022_10_15_55
CIP_01_12_2022_10_15_58
CIP_01_12_2022_10_15_59
CIP_01_12_2022_10_16_20
CIP_01_12_2022_10_16_25
CIP_02_12_2022_10_15_20
In the above, I have to delete between two dates. For example I have to delete between these dates 01-12-2022 00:00:00 to 01-12-2022 11:59:59 delete all tables except 2nd December 2022 table.
I think you can use a cursor in order to do that:
DECLARE table_cursor CURSOR FOR
SELECT name
FROM sys.tables
WHERE name LIKE 'CIP_%'
Then do loop on it.
EXEC('DROP TABLE ' + #<yourFetchVariable>)
Below is an example that uses a strict pattern matching for the dated tables and extracts the date value from the name for use in the date range criteria (a single date per your sample data).
DECLARE
#StartDate date = '20221201' --ISO 8601 date format to avoid ambiguity
, #EndDate date = '20221201'
, #DropScript nvarchar(MAX);
WITH date_tables AS (
SELECT name AS TableName, TRY_CAST(SUBSTRING(name, 11, 4)+SUBSTRING(name, 8, 2)+SUBSTRING(name, 5, 2) AS date) AS TableDate
FROM sys.tables
WHERE name LIKE N'CIP[_][0-9][0-9][_][0-9][0-9][_][0-9][0-9][0-9][0-9][_][0-9][0-9][_][0-9][0-9][_][0-9][0-9]'
)
SELECT #DropScript = STRING_AGG(N'DROP TABLE ' + TableName, N';') + N';'
FROM date_tables
WHERE TableDate BETWEEN #StartDate AND #EndDate;
EXEC sp_executesql #DropScript;
GO
I've inherited a query that has parameters which specify pulls data for a single desired month. The extract then gets manually added to previous month's extract in Excel. I'd like to eliminate the manual portion by adjusting the existing query to iterate across all months greater than a given base month, then (if this is what makes most sense) unioning the individual "final" outputs.
My attempt was to add the entire block of code for each specific month to the existing code, and then run it together. The idea was that I'd just paste in a new block each new month. I knew this was very inefficient, but I don't have the luxury of learning how to do it efficiently, so if it worked I'd be happy.
I ran into problems because the existing query has two subqueries which then are used to create a final table, and I couldn't figure out how to retain the final table at the end of the code so that it could be referenced in a union later (fwiw, I was attempting to use a Select Into for that final table).
with eligibility_and_customer_type AS
(SELECT DISTINCT ON(sub_id, mbr_sfx_id)
sub_id AS subscriber_id
, mbr_sfx_id AS member_suffix_id
, src_mbr_key
, ctdv.cstmr_typ_cd
, gdv.grp_name
FROM adw_common.cstmr_typ_dim_vw ctdv
JOIN adw_common.mbr_eligty_by_mo_fact_vw
ON ctdv.cstmr_typ_key = mbr_eligty_by_mo_fact_vw.cstmr_typ_key
AND mbr_eligty_yr = '2018'
AND mbr_eligty_mo = '12'
JOIN adw_common.prod_cat_dim_vw
ON prod_cat_dim_vw.prod_cat_key = mbr_eligty_by_mo_fact_vw.prod_cat_key
AND prod_cat_dim_vw.prod_cat_cd = 'M'
JOIN adw_common.mbr_dim_abr
ON mbr_eligty_by_mo_fact_vw.mbr_key = mbr_dim_abr.mbr_key
JOIN consumer.facets_xref_abr fxf
ON mbr_dim_abr.src_mbr_key = fxf.source_member_key
JOIN adw_common.grp_dim_vw gdv
ON gdv.grp_key=mbr_eligty_by_mo_fact_vw.grp_key),
facets_ip as
(select distinct cl.meme_ck
FROM gpgen_cr_ai.cmc_clcl_claim_abr cl
/* LEFT JOIN gpgen_cr_ai.cmc_clhp_hosp_abr ch
ON cl.clcl_id = ch.clcl_id*/
LEFT JOIN gpgen_cr_ai.cmc_cdml_cl_line cd
ON cl.clcl_id = cd.clcl_id
WHERE cd.pscd_id = '21'
/*AND ch.clcl_id IS NULL*/
AND cl.clcl_cur_sts NOT IN ('91','92')
AND cl.clcl_low_svc_dt >= '20181201'
and cl.clcl_low_svc_dt <= '20181231'
group by 1)
select distinct c.meme_ck,
e.cstmr_typ_cd,
'201812' as Yearmo
from facets_ip c
left join eligibility_and_customer_type e
on c.meme_ck = e.src_mbr_key;
The code above has date parameters that get updated when necessary.
The final output would be a version of the final table created above, but with results corresponding to, say, 201801 - present.
If you provide:
DDL of the underlying tables
Sample Data of the underlying tables
Expected resultset
DBMS you are using
, then one would be able to provide the best solution here.
Without knowing them, and as you said you only care about dynamically looping through each month, here is one way you can utilize your code to loop it through in SQL Server. Please fill the variable #StartDate and #EndDate values and provide proper datatype for meme_ck and cstmr_typ_cd.
IF OBJECT_ID ('tempdb..#TempTable', N'U') IS NOT NULL
BEGIN
DROP TABLE #TempTable
END
CREATE TABLE #TempTable
(
meme_ck <ProvideProperDataTypeHere>
,cstmr_typ_cd <ProvideProperDataTypeHere>
,Yearmo VARCHAR(10)
)
DECLARE #StartDate DATE = '<Provide the first day of the start month>'
DECLARE #EndDate DATE = '<Provide the end date inclusive>'
WHILE #StartDate <= #EndDate
BEGIN
DECLARE #MonthEndDate DATE = CASE WHEN DATEADD(DAY, -1, DATEADD(MONTH, 1, #StartDate)) <= #EndDate THEN DATEADD(DAY, -1, DATEADD(MONTH, 1, #StartDate)) ELSE #EndDate END
DECLARE #MonthYear VARCHAR(6) = LEFT(CONVERT(VARCHAR(8), #StartDate, 112), 6)
--This is your code which I am not touching without not knowing any detail about it. Just feeding the variables to make it dynamic
;with eligibility_and_customer_type AS
(SELECT DISTINCT ON(sub_id, mbr_sfx_id)
sub_id AS subscriber_id
, mbr_sfx_id AS member_suffix_id
, src_mbr_key
, ctdv.cstmr_typ_cd
, gdv.grp_name
FROM adw_common.cstmr_typ_dim_vw ctdv
JOIN adw_common.mbr_eligty_by_mo_fact_vw
ON ctdv.cstmr_typ_key = mbr_eligty_by_mo_fact_vw.cstmr_typ_key
AND mbr_eligty_yr = CAST(YEAR(#StartDate) AS VARCHAR(10)) -- NO need to cast if mbr_eligty_yr is an Integer
AND mbr_eligty_mo = CAST(MONTH(#StartDate) AS VARCHAR(10)) -- NO need to cast if mbr_eligty_yr is an Integer
JOIN adw_common.prod_cat_dim_vw
ON prod_cat_dim_vw.prod_cat_key = mbr_eligty_by_mo_fact_vw.prod_cat_key
AND prod_cat_dim_vw.prod_cat_cd = 'M'
JOIN adw_common.mbr_dim_abr
ON mbr_eligty_by_mo_fact_vw.mbr_key = mbr_dim_abr.mbr_key
JOIN consumer.facets_xref_abr fxf
ON mbr_dim_abr.src_mbr_key = fxf.source_member_key
JOIN adw_common.grp_dim_vw gdv
ON gdv.grp_key=mbr_eligty_by_mo_fact_vw.grp_key),
facets_ip as
(select distinct cl.meme_ck
FROM gpgen_cr_ai.cmc_clcl_claim_abr cl
/* LEFT JOIN gpgen_cr_ai.cmc_clhp_hosp_abr ch
ON cl.clcl_id = ch.clcl_id*/
LEFT JOIN gpgen_cr_ai.cmc_cdml_cl_line cd
ON cl.clcl_id = cd.clcl_id
WHERE cd.pscd_id = '21'
/*AND ch.clcl_id IS NULL*/
AND cl.clcl_cur_sts NOT IN ('91','92')
AND cl.clcl_low_svc_dt BETWEEN #StartDate AND #MonthEndDate
group by 1)
INSERT INTO #TempTable
(
meme_ck
,cstmr_typ_cd
,Yearmo
)
select distinct c.meme_ck,
e.cstmr_typ_cd,
#MonthYear as Yearmo
from facets_ip c
left join eligibility_and_customer_type e
on c.meme_ck = e.src_mbr_key;
SET #StartDate = DATEADD(MONTH, 1, #StartDate)
END
SELECT * FROM #TempTable;
I don't have enough information on your tables to really create an optimal solution. The solutions I am providing just have a single parameter (table name) and for your solution, you will need to pass in an additional parameter for the date filter.
The idea of "looping" is not something you'll need to do in Greenplum. That is common for OLTP databases like SQL Server or Oracle that can't handle big data very well and have to process smaller amounts at a time.
For these example solutions, a table is needed with some data in it.
CREATE TABLE public.foo
(id integer,
fname text,
lname text)
DISTRIBUTED BY (id);
insert into foo values (1, 'jon', 'roberts'),
(2, 'sam', 'roberts'),
(3, 'jon', 'smith'),
(4, 'sam', 'smith'),
(5, 'jon', 'roberts'),
(6, 'sam', 'roberts'),
(7, 'jon', 'smith'),
(8, 'sam', 'smith');
Solution 1: Learn how functions work in the database. Here is a quick example of how it would work.
Create a function that does the Create Table As Select (CTAS) where you pass in a parameter.
Note: You can't execute DDL statements in a function directly so you have to use "EXECUTE" instead.
create or replace function fn_test(p_table_name text) returns void as
$$
declare
v_sql text;
begin
v_sql :='drop table if exists ' || p_table_name;
execute v_sql;
v_sql := 'create table ' || p_table_name || ' with (appendonly=true, compresstype=quicklz) as
with t as (select * from foo)
select * from t
distributed by (id)';
execute v_sql;
end;
$$
language plpgsql;
Execute the function with a simple select statement.
select fn_test('foo3');
Notice how I pass in a table name that will be created when you execute the function.
Solution 2: Use psql variables
Create a sql file name "test.sql" with the following contents.
drop table if exists :p_table_name;
create table :p_table_name with (appendonly=true, compresstype=quicklz) as
with t as (select * from foo)
select * from t
distributed by (id);
Next, you execute psql and pass in the variable p_table_name.
psql -f test.sql -v p_table_name=foo4
psql:test.sql:1: NOTICE: table "foo4" does not exist, skipping
DROP TABLE
SELECT 8
I have a COTS application database that creates a new table every week that follows a specific naming convention and always contains the same columns. I have written a query to select the tables within a certain time range:
DECLARE #today VARCHAR(8)
SET #today = CONVERT(VARCHAR(8),GETDATE(),112)
DECLARE #monthold VARCHAR(8)
SET #monthold = CONVERT(VARCHAR(8),dateadd(day,-31,#today),112)
SELECT name
FROM sys.tables
WHERE name <= 'Event_Audit'+ #today AND name >= 'Event_Audit'+ #monthold
AND name like 'Event_Audit%'
ORDER BY name DESC
Now I am looking for a way to have a query call each of the tables that is selected to aggregate the data. This will be used for SSRS reporting.
Something like this where table1, table2, etc. through all the included tables will get populated (whether it is 4, 5, or more):
SELECT *
FROM table1
UNION ALL
SELECT *
FROM table2
UNION ALL
...
ORDER BY EVENT_DATE DESC
I can't create views, new tables, or make any changes to existing tables.
Poor design will lead to hard work every time, this one is no exception. I'm guessing by now it's too late to give up the idea of creating the same table over and over again for each month.
You do not need to create a new or temporary table to execute sp_executeSql, you only need to generate the sql script:
DECLARE #today char(8) = CONVERT(char(8),GETDATE(),112),
#monthold char(8) = CONVERT(char(8),dateadd(day,-31,GETDATE()),112),
#sql nvarchar(max) = ''
SELECT #Sql = #Sql + ' UNION ALL SELECT * FROM ' + name
FROM sys.tables
WHERE name <= 'Event_Audit'+ #today AND name >= 'Event_Audit'+ #monthold
AND name like 'Event_Audit%'
ORDER BY name DESC
SET #SQL = STUFF(#SQL, 1, 11, '') + -- remove the first UNION ALL
' ORDER BY EVENT_DATE DESC' -- add the ORDER BY
PRINT #Sql
--EXEC sp_executeSql #Sql
Once you've printed the #Sql and checked it's ok, unremark the EXEC line and run your script.
I have a table with a string in some columns values that tells me if I should delete the row....however this string needs some parsing to understand whether to delete or not.
What is the string: it tells me the recurrence of meetings eg everyday starting 21st march for 10 meetings.
My table is a single column called recurrence:
Recurrence
-------------------------------
daily;1;21/03/2015;times;10
daily;1;01/02/2016;times;8
monthly;1;01/01/2016;times;2
weekly;1;21/01/2016;times;4
What to do: if the meetings are finished then remove the row.
The string is of the following format
<frequency tag>;<frequency number>;<start date>;times;<no of times>
For example
daily;1;21/03/2016;times;10
everyday starting 21st march, for 10 times
Does anybody know how I would calculate if the string indicates all meetings are in past? I want a select statement that tells me if the recurrence values are in past - true or false
I added one string ('weekly;1;21/05/2016;times;4') that definitely must not be deleted to show some output. At first try to add to temp table `#table1' all data from your table and check if all is deleted well.
DECLARE #table1 TABLE (
Recurrence nvarchar(max)
)
DECLARE #xml xml
INSERT INTO #table1 VALUES
('daily;1;21/03/2016;times;10'),
('daily;1;21/03/2015;times;10'),
('daily;1;01/02/2016;times;8'),
('monthly;1;01/01/2016;times;2'),
('weekly;1;21/01/2016;times;4'),
('weekly;1;21/05/2016;times;4')
SELECT #xml= (
SELECT CAST('<s><r>' + REPLACE(Recurrence,';','</r><r>') + '</r><r>'+ Recurrence+'</r></s>' as xml)
FROM #table1
FOR XML PATH ('')
)
;WITH cte as (
SELECT t.v.value('r[1]','nvarchar(10)') as how,
t.v.value('r[2]','nvarchar(10)') as every,
CONVERT(date,t.v.value('r[3]','nvarchar(10)'),103) as since,
t.v.value('r[4]','nvarchar(10)') as what,
t.v.value('r[5]','int') as howmany,
t.v.value('r[6]','nvarchar(max)') as Recurrence
FROM #xml.nodes('/s') as t(v)
)
DELETE t
FROM #table1 t
LEFT JOIN cte c ON c.Recurrence=t.Recurrence
WHERE
CASE WHEN how = 'daily' THEN DATEADD(day,howmany,since)
WHEN how = 'weekly' THEN DATEADD(week,howmany,since)
WHEN how = 'monthly' THEN DATEADD(month,howmany,since)
ELSE NULL END < GETDATE()
SELECT * FROM #table1
Output:
Recurrence
-----------------------------
weekly;1;21/05/2016;times;4
(1 row(s) affected)
I have a table in an Access DB that has columns for each of the 12 months of the year. The columns have names of "1" to "12". That is to say, the names of the columns are just the number of the month it represents. The columns contain numbers as data, and I need to sum the columns for the months remaining in the year. For example, right now we're in September, so I'd need the SELECT clause to sum the values in columns (months) 9 through 12. This must be able to dynamically sum up the relevant months, so that next month (Oct) Sep will be excluded and only 10 through 12 are summed.
How can I reference the name of the column in the SELECT clause to be able to perform a test on it. I need something like the following:
IIf(Table1.[1].ColumnName >= Month(Now), Table1.[1], 0)
+ IIf(Table1.[2].ColumnName >= Month(Now), Table1.[2], 0)
...
+ IIf(Table1.[12].ColumnName >= Month(Now), Table1.[12], 0)
This would be one approach, but, in passing, if there's a better way to do this, please let me know as well.
I've seen other posts on SO that discuss returning all column names for a table. That is not what I need here. I need to return the column name and perform tests on it within a SELECT clause. Thanks.
EDIT
I understand that this structure (having the data across 12 different columns) is not the greatest. This is not a new DB that I'm setting up. It is an old DB that I've inherited, so I can't make changes to the structure.
Not sure I understood your question, but it would seem the major issue is the fact that you store values in different columns instead of something like .
One solution could be to use union in a subquery to make the columns available to the outer select and use that to filter, sum and group by.
If you have a table looking like:
id (number, primary key), 1..12 (number, month values)
Then a query like this should work (this is for month 1-5 only):
SELECT id, SUM(MonthValue) AS SumFutureMonths
FROM (
SELECT 1 AS MonthNo, id, YourTable.[1] AS MonthValue FROM YourTable
union all
SELECT 2 AS MonthNo, id, YourTable.[2] AS MonthValue FROM YourTable
union all
SELECT 3 AS MonthNo, id, YourTable.[3] AS MonthValue FROM YourTable
union all
SELECT 4 AS MonthNo, id, YourTable.[4] AS MonthValue FROM YourTable
union all
SELECT 5 AS MonthNo, id, YourTable.[5] AS MonthValue FROM YourTable
)
WHERE MonthNo > MONTH(NOW())
GROUP BY id
Whether it would perform well I can't say - it depends on your data, but the retrieving all data in the table for a union can be a costly operation. Anyway, please try it out to see if it works.
I don't know if this works in Access; but in MS SQL you can write a string and execute it..
For example
#sqlStr = "Select 9, 10, 11, 12 from table"
Exec(#sqlStr)
And a string would be fairly easy to manipulate to contain relevant info..
But another question - why would you have all your data in each column instead of different rows? That way you could really easily get the data you want and show it the way you would want to..
UPDATE:
DECLARE #i int
SET #i = MONTH(getdate())
DECLARE #str nvarchar(max)
SET #str = 'SELECT '
WHILE #i <= 12
BEGIN
SET #str = #str + cast(#i as nvarchar(max))+ ','
SET #i = #i + 1
END
SET #str = LEFT(#str, len(#str) - 1) + ' FROM TABLE'
Exec(#str)