SQL Server - Manipulate data and return new results - sql

Is there any way to do something in SQL Server to manipulate data like manipulating arrays in any other programming language?
I have one SQL query that returns 3 columns, "dt_ref" (date), "vlr_venda" (float) and "qt_parcelas" (int)
Basically, I need to do something like this:
- When field "qt_parcelas" is higher than 1, I need to do a "loop" with this row and generate 3 rows.
So, I need to divide the field "vlr_venda" by field "qt_parcelas", and use the field "dt_ref" as reference for the date start and increment month in the date field for the value of "qt_parcelas"
For example, if my query returns these structure:
| dt_ref | vlr_venda | qt_parcelas |
-------------------------------------
|20180901 | 3000 | 3 |
I need to do something to return this:
| dt_ref | vlr_venda |
----------------------
|20180901 | 1000 |
|20181001 | 1000 |
|20181101 | 1000 |
Is it possible to do it in SQL Server?
I've searched for something like this but haven't found anything useful...
Any ideas?

You can use a recursive CTE: Sql Fiddle
with cte as (
select dt_ref, vlr_venda / qt_parcelas as new_val, qt_parcelas, 1 as num
from t
union all
select dateadd(month, 1, dt_ref), new_val, qt_parcelas, num + 1
from cte
where num < qt_parcelas
)
select dt_ref, new_val
from cte;
As written, this will work for up to 100 months. You need to add OPTION (MAXRECURSION 0) for longer periods.

Instead of using a rCTE you could use a tally table. If you have much larger numbers than 3 this'll probably be far be efficient:
WITH N AS(
SELECT n
FROM (VALUES(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL)) N(n)),
Tally AS (
SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) I
FROM N N1 --10 rows
CROSS JOIN N N2 --100 rows
--CROSS JOIN N N3 --Keep adding more CROSS JOINs to create more rows
),
VTE AS (
SELECT CONVERT(date,V.dt_ref) AS dt_ref,
V.vlr_venda,
V.qt_parcelas
FROM (VALUES('20180901',3000,3),
('20181001',12000,6)) V(dt_ref,vlr_venda ,qt_parcelas))
SELECT DATEADD(MONTH,T.I,V.dt_ref),
CONVERT(decimal(10,4),V.vlr_venda / (V.qt_parcelas * 1.0)) --incase you need decimal points
FROM VTE V
JOIN Tally T ON V.qt_parcelas >= T.I;

I developed a software to generate tickets and i had a similar experience than you. I have tried CURSORS and recursive CTE and they all took something like 50 minutes when creating tickets for clients
I used this function to replicate my clients and generate my tickets
/****** Object: UserDefinedFunction [dbo].[NumbersTable] Script Date: 28/09/2018 10:51:25 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE FUNCTION [dbo].[NumbersTable] (
#fromNumber int,
#toNumber int,
#byStep int
)
RETURNS #NumbersTable TABLE (i int)
AS
BEGIN
WITH CTE_NumbersTable AS (
SELECT #fromNumber AS i
UNION ALL
SELECT i + #byStep
FROM CTE_NumbersTable
WHERE
(i + #byStep) <= #toNumber
)
INSERT INTO #NumbersTable
SELECT i FROM CTE_NumbersTable OPTION (MAXRECURSION 0)
RETURN;
END
GO
Then you can use a
CROSS APPLY dbo.NumbersTable(1,qt_parcelas ,1);
To generate your rows
Believe me this way is more efficient and when dealing with large amount of data (something like 8 to 10 million rows) it takes something like 2 minutes instead of 40

Related

Redshift: Generate a sequential range of numbers

I'm currently migrating PostgreSQL code from our existing DWH to new Redshift DWH and few queries are not compatible.
I have a table which has id, start_week, end_week and orders_each_week in a single row. I'm trying to generate a sequential series between the start_week and end_week so that I separate rows for each week between the give timeline.
Eg.,
This how it is present in the table
+----+------------+----------+------------------+
| ID | start_week | end_week | orders_each_week |
+----+------------+----------+------------------+
| 1 | 3 | 5 | 10 |
+----+------------+----------+------------------+
This is how I want to have it
+----+------+--------+
| ID | week | orders |
+----+------+--------+
| 1 | 3 | 10 |
+----+------+--------+
| 1 | 4 | 10 |
+----+------+--------+
| 1 | 5 | 10 |
+----+------+--------+
The code below is throwing error.
SELECT
id,
generate_series(start_week::BIGINT, end_week::BIGINT) AS demand_weeks
FROM client_demand
WHERE createddate::DATE >= '2021-01-01'
[0A000][500310] Amazon Invalid operation: Specified types or functions (one per INFO message) not supported on Redshift tables.;
[01000] Function "generate_series(bigint,bigint)" not supported.
So basically I am trying to find a sequential series between two numbers and I couldn't find any solution and any help here is really appreciated. Thank you
Gordon Linoff has shown a very common method for doing this and this approach has the advantage that the process isn't generating "rows" that don't already exist. This can make this faster than approaches that generate data on the fly. However, you need to have a table with about the right number of rows laying around and this isn't always the case. He also shows that this number series needs to be cross joined with your data to perform the function you need.
If you need to generate a large number of numbers in a series not using an existing table there are a number of ways to do this. Here's my go to approach:
WITH twofivesix AS (
SELECT
p0.n
+ p1.n * 2
+ p2.n * POWER(2,2)
+ p3.n * POWER(2,3)
+ p4.n * POWER(2,4)
+ p5.n * POWER(2,5)
+ p6.n * POWER(2,6)
+ p7.n * POWER(2,7)
as n
FROM
(SELECT 0 as n UNION SELECT 1) p0,
(SELECT 0 as n UNION SELECT 1) p1,
(SELECT 0 as n UNION SELECT 1) p2,
(SELECT 0 as n UNION SELECT 1) p3,
(SELECT 0 as n UNION SELECT 1) p4,
(SELECT 0 as n UNION SELECT 1) p5,
(SELECT 0 as n UNION SELECT 1) p6,
(SELECT 0 as n UNION SELECT 1) p7
),
fourbillion AS (
SELECT (a.n * POWER(256, 3) + b.n * POWER(256, 2) + c.n * 256 + d.n) as n
FROM twofivesix a,
twofivesix b,
twofivesix c,
twofivesix d
)
SELECT ...
This example makes a whole bunch of numbers (4B) but you can extend or reduce the number in the series by changing the number of times the tables are cross joined and by adding where clauses (as Gordon Linoff did). I don't expect you need a list anywhere close to this long but wanted to show how this can be used to make series that are very long. (You can also write with in base 10 if that makes more sense to you.)
So if you have a table with a more rows that you need number then this can be the fastest method but if you don't have such a table or table lengths vary over time you may want this pure SQL approach.
Among the many Postgres features that Redshift does not support is generate_series() (except on the master node). You can generate one yourself.
If you have a table with enough rows in Redshift, then I find that this approach works:
with n as (
select row_number() over () - 1 as n
from client_demand cd
)
select cd.id, cd.start_week + n.n as week, cd.orders_each_week
from client_demand cd join
n
on n.n <= (end_week - start_week);
This assumes that you have a table with enough rows to generate enough numbers for the on clause. If the table is really big, then add something like limit 100 in the n CTE to limit the size.
If there are only a handful of values, you can use:
select 0 as n union all
select 1 as n union all
select 2 as n

Generate SQL rows

Given a number of types and a number of occurrences per type, I would like to generate something like this in T-SQL:
Occurrence | Type
-----------------
0 | A
1 | A
0 | B
1 | B
2 | B
Both the number of types and the number of occurrences per type are presented as values in different tables.
While I can do this with WHILE loops, I'm looking for a better solution.
Thanks!
This works with a number-table which i would use.
SELECT Occurrence = ROW_NUMBER() OVER (PARTITION BY Type ORDER BY Type) - 1
, Type
FROM Numbers num
INNER JOIN #temp1 t
ON num.n BETWEEN 1 AND t.Occurrence
Tested with this sample data:
create table #temp1(Type varchar(10),Occurrence int)
insert into #temp1 VALUES('A',2)
insert into #temp1 VALUES('B',3)
How to create a number-table? http://sqlperformance.com/2013/01/t-sql-queries/generate-a-set-1
If you have a table with the columns type and num, you have two approaches. One way is to use recursive CTEs:
with CTE as (
select type, 0 as occurrence, num
from table t
union all
select type, 1 + occurrence, num
from cte
where occurrence + 1 < num
)
select cte.*
from cte;
You may have to set the MAXRECURSION option, if the number exceeds 100.
The other way is to join in a numbers table. SQL Server uses spt_values for this purpose:
select s.number - 1 as occurrence, t.type
from table t join
spt_values s
on s.number <= t.num ;

How to split the string value in one column and return the result table

Assume we have the following table:
id name member
1 jacky a;b;c
2 jason e
3 kate i;j;k
4 alex null
Now I want to use the sql or t-sql to return the following table:
1 jacky a
1 jacky b
1 jacky c
2 jason e
3 kate i
......
How to do that?
I'm using the MSSQL, MYSQL and Oracle database.
This is the shortest and readable string-to-rows splitter one could devise, and could be faster too.
Use case of choosing pure CTE instead of function, e.g. when you're not allowed to create a function on database :-)
Creating rows generator via function(which could be implemented by using loop or via CTE too) shall still need to use lateral joins(DB2 and Sybase have this functionality, using LATERAL keyword; In SQL Server, this is similar to CROSS APPLY and OUTER APPLY) to ultimately join the splitted rows generated by a function to the main table.
Pure CTE approach could be faster than function approach. The speed metrics lies in profiling though, just check the execution plan of this compared to other solutions if this is indeed faster:
with Pieces(theId, pn, start, stop) AS
(
SELECT id, 1, 1, charindex(';', member)
from tbl
UNION ALL
SELECT id, pn + 1, stop + 1, charindex(';', member, stop + 1)
from tbl
join pieces on pieces.theId = tbl.id
WHERE stop > 0
)
select
t.id, t.name,
word =
substring(t.member, p.start,
case WHEN stop > 0 THEN p.stop - p.start
ELSE 512
END)
from tbl t
join pieces p on p.theId = t.id
order by t.id, p.pn
Output:
ID NAME WORD
1 jacky a
1 jacky b
1 jacky c
2 jason e
3 kate i
3 kate j
3 kate k
4 alex (null)
Base logic sourced here: T-SQL: Opposite to string concatenation - how to split string into multiple records
Live test: http://www.sqlfiddle.com/#!3/2355d/1
Well... let me first introduce you to Adam Machanic who taught me about a Numbers table. He's also written a very fast split function using this Numbers table.
http://dataeducation.com/counting-occurrences-of-a-substring-within-a-string/
After you implement a Split function that returns a table, you can then join against it and get the results you want.
IF OBJECT_ID('dbo.Users') IS NOT NULL
DROP TABLE dbo.Users;
CREATE TABLE dbo.Users
(
id INT IDENTITY NOT NULL PRIMARY KEY,
name VARCHAR(50) NOT NULL,
member VARCHAR(1000)
)
GO
INSERT INTO dbo.Users(name, member) VALUES
('jacky', 'a;b;c'),
('jason', 'e'),
('kate', 'i;j;k'),
('alex', NULL);
GO
DECLARE #spliter CHAR(1) = ';';
WITH Base AS
(
SELECT 1 AS n
UNION ALL
SELECT n + 1
FROM Base
WHERE n < CEILING(SQRT(1000)) --generate numbers from 1 to 1000, you may change it to a larger value depending on the member column's length.
)
, Nums AS --Numbers Common Table Expression, if your database version doesn't support it, just create a physical table.
(
SELECT ROW_NUMBER() OVER(ORDER BY (SELECT 0)) AS n
FROM Base AS B1 CROSS JOIN Base AS B2
)
SELECT id,
SUBSTRING(member, n, CHARINDEX(#spliter, member + #spliter, n) - n) AS element
FROM dbo.Users
JOIN Nums
ON n <= DATALENGTH(member) + 1
AND SUBSTRING(#spliter + member, n, 1) = #spliter
ORDER BY id
OPTION (MAXRECURSION 0); --Nums CTE is generated recursively, we don't want to limit recursion count.

Repeating Record Sequence using SQL

This could easily be done using code, but I wondered if it could be done at the database level using SQL Server (2008).
I have a table similar to below:
CROP_ID YEAR_ PRODUCTION
1 1 0
1 2 300
1 3 500
2 1 100
2 2 700
I want to be able to run a query to repeat this for n number of years, per crop type e.g.
CROP_ID YEAR_ PRODUCTION
1 1 0
1 2 300
1 3 500
1 4 0
1 5 300
1 6 500
etc.
I'm not sure of the best approach, I presume I would need a SP and pass in a year variable, and use a loop construct? However the exact syntax escapes me. Any help appreciated.
Update
Sorry for not providing all the information in my original post. The table will allow for multiple crop types, and for Produciton values to be updated so Case statements with fixed variables are not really suitable. Apologies for not being clearer.
Update
With the TVF answer I used the following modified SQL to select by CropType for 20 years.
select top 20 b.CROP_ID,
YEAR_ = n.num * (select count() from MyBaseTable where CROP_ID = 3) + b.YEAR,
b.PRODUCTION from MyBaseTable b, dbo.fnMakeNRows(20) n
where CROP_ID = 3
You can do this in standard SQL without creating a stored procedure or using temp tables. The example below will do this for 12 years. You can extend it out to any number of years:
insert into CropYield
(CropID, Year_, Production)
Select 1, a.a + (10 * b.a),
case (a.a + (10 * b.a)) % 3
when 0 then 500
when 1 then 0
when 2 then 300
end
from (Select 0 as a union select 1 union select 2 union select 3 union select 4 union select 5 union select 6 union select 7 union select 8 union select 9) as a
cross join (Select 0 as a union select 1 union select 2 union select 3 union select 4 union select 5 union select 6 union select 7 union select 8 union select 9) as b
where a.a + (10 * b.a) between 1 and 12
You could use a table-valued-function instead of a stored proc, which gives a little more flexibility for what you do with the result (as it can be selected from directly, inserted into another table, joined to other tables, etc).
You could also make this more generic by having a TVF generate N rows (with a number from 0 to N-1 on each row) and then use some simple expressions to generate the columns you need from this. I have found such a TVF to be useful in a variety of situations.
If you need to generate more complicated data than you can with 0..N-1 and simple expressions, then you should create a TVF dedicated to your specific needs.
The following example shows how a generic TVF could be used to generate the data you ask for:
create function fnMakeNRows (#num as integer)
returns #result table (num integer not null) as
begin
if #num is null or #num = 0
begin
return
end
declare #n as integer
set #n = 0
while #n < #num
begin
insert into #result values (#n)
set #n = #n + 1
end
return
end
go
select
CROP_ID = 1,
YEAR_ = num,
PRODUCTION = case num % 3 when 0 then 0 when 1 then 300 else 500 end
from dbo.fnMakeNRows(100000)
You can also use this to duplicate rows in an existing table (which I think is more like what you want). For example, assuming base_table contains the three rows at the beginning of your question, you can turn the 3 rows into 60 rows using the following:
select
b.CROP_ID,
YEAR_ = n.num * (select count(*) from base_table) + b.YEAR_,
b.PRODUCTION
from base_table b, dbo.fnMakeNRows(20) n
This (hopefully) shows the utility of a generic fnMakeNRows function.
A common trick to produce this kind of data without the need of a stored procedure is with the use of a table of constants. Because such a table can be of generic use, it can be created with say all the integers between 1 and 100 or even 1 and 1,000 depending on usage.
For exmaple
CREATE TABLE tblConstNums
( I INT )
INSERT INTO tblConstNums VALUES (1)
INSERT INTO tblConstNums VALUES (2)
INSERT INTO tblConstNums VALUES (3)
INSERT INTO tblConstNums VALUES (4)
INSERT INTO tblConstNums VALUES (5)
-- ...
INSERT INTO tblConstNums VALUES (1000)
The the solution can be written declaratively (without requiring Stored Procedure or more generally procedural statements:
SELECT CROP_ID, YEAR_ * I, PRODUCTION
FROM myCropTable T
JOIN tblConstNums C on 1=1
WHERE I in (1, 2, 3)
order by CROP_ID, YEAR_ * I, PRODUCTION
Note that the table of constants may include several columns for commonly used cases. For example, and even though many of these can be expressed as mathematical expressions of numbers in a basic 0 to n sequence, one can have a column with only even number, another one with odd numbers, another one with multiples of 5 etc. Also if it small enough, no indexes are needed on a table of constants but these may become useful on a bigger on.
use this one:
WITH tn (n) as
(
SELECT 0
UNION ALL
SELECT n+1
FROM tn
WHERE tn.n < 10
)
SELECT DISTINCT t.CROP_ID, t.YEAR_ + (3*tn.n), t.PRODUCTION
FROM table t, tn
/*
WHERE tn.n < 10 ==> you will get 1 -> (10*3) + 3 = 33
*/

SQL return multiple rows from one record

This is the opposite of reducing repeating records.
SQL query to create physical inventory checklists
If widget-xyz has a qty of 1 item return 1 row, but if it has 5, return 5 rows etc.
For all widgets in a particular warehouse.
Previously this was handled with a macro working through a range in excel, checking the qty column. Is there a way to make a single query instead?
The tables are FoxPro dbf files generated by an application and I am outputting this into html
Instead of generating an xml string and using xml parsing functions to generate a counter as Nestor has suggested, you might consider joining on a recursive CTE as a counter, as LukLed has hinted to:
WITH Counter AS
(
SELECT 0 i
UNION ALL
SELECT i + 1
FROM Counter
WHERE i < 100
),
Data AS
(
SELECT 'A' sku, 1 qty
UNION
SELECT 'B', 2
UNION
SELECT 'C', 3
)
SELECT *
FROM Data
INNER JOIN Counter ON i < qty
According to query analyzer, this query is much faster than the xml pseudo-table. This approach also gives you a recordset with a natural key (sku, i).
There is a default recursion limit of 100 in MSSQL that will restrict your counter. If you have quantities > 100, you can either increase this limit, use nested counters, or create a physical table for counting.
For SQL 2005/2008, take a look at
CROSS APPLY
What I would do is CROSS APPLY each row with a sub table with as many rows as qty has. A secondary question is how to create that sub table (I'd suggest to create an xml string and then parse it with the xml operators)
I hope this gives you a starting pointer....
Starting with
declare #table table (sku int, qty int);
insert into #table values (1, 5), (2,4), (3,2);
select * from #table;
sku qty
----------- -----------
1 5
2 4
3 2
You can generate:
with MainT as (
select *, convert(xml,'<table>'+REPLICATE('<r></r>',qty)+'</table>') as pseudo_table
from #table
)
select p.sku, p.qty
from MainT p
CROSS APPLY
(
select p.sku from p.pseudo_table.nodes('/table/r') T(row)
) crossT
sku qty
----------- -----------
1 5
1 5
1 5
1 5
1 5
2 4
2 4
2 4
2 4
3 2
3 2
Is that what you want?
Seriously dude... next time put more effort writing your question. It's impossible to know exactly what you are looking for.
You can use table with number from 1 to max(quantity) and join your table by quantity <= number. You can do it in many ways, but it depends on sql engine.
You can do this using dynamic sql.