I'm a longtime SAS programmer and we're looking at moving our system from SAS to another platform. I've only a very basic knowledge of SQL the marketing folks talk about using SQL a lot but I wonder how it might do some things we need done. For instance, we have files with up to 50 million rows of vaccination records for each vaccine that was administered to a patient. Some vaccines are actually a combination vaccine that represent 2-4 different types of vaccines. The type of vaccine is based on the value of CVX. Using a do-loop it's fairly simple to do this in SAS, but I've no idea of how it might be done in SQL. It's be safe to assume that we have all the CVX codes in a table with 1 to 4 vaccine types that need to be generated. But how would you do it in SQL?
Thanks,
Steve
I don't know anything about your schema, but it sounds like you're looking to split multiple columns into rows. You can use either CROSS APPLY or the UNPIVOT operator. Here is an example that takes a contrived test table and splits it into separate rows per key:
create table #Test
(
Test_Key int identity(1,1) primary key clustered,
Test_A int,
Test_B int,
Test_C int
)
declare #n int
set #n = 1
while #n < 10000
begin
insert into #Test (Test_A, Test_B, Test_C)
select #n * 5 + 1, #n * 5 + 2, #n * 5 + 3
set #n = #n + 1
end
select * from #Test
-- this example converts the columns into rows using CROSS APPLY
-- this may be slightly less expensive than the UNPIVOT example below
select
F_Key,
F_Value
from #Test
cross apply
(
values
(Test_Key, Test_A), -- 1st row is Test_A
(Test_Key, Test_B), -- 2nd row is Test_B
(Test_Key, Test_C) -- 3rd row is Test_C
) as F(F_Key, F_Value)
-- this example converts the columns into rows using the UNPIVOT operator
select
Test_Key, TestKey
from #Test
unpivot
(TestKey for Test_Type in (Test_A, Test_B, Test_C)) as C
drop table #Test
Related
I have a table in MS SQL Server that contains multiple TEXT fields that can have very long strings (from 0 characters to 100,000+ characters).
I'd like to create a view (or a stored proc that populates a reporting table) that prepares this data for export to Excel, which has a certain character limit allowable per cell (32,767 chars).
It's relatively trivial to write a query to truncate the fields after a certain number of characters, but I'd like to create new rows containing the text that would be truncated.
Example - Row 1, Col1 and Col3 contains text that is wrapped into 2 rows.
ID | COL1 | COL 2 | COL 3 |
1 AAAAAA BBBBBBB CCCCCC
1 AAA CC
2 XX YY ZZ
You can try something along this:
A mockup table to simulate your issue
DECLARE #tbl TABLE(ID INT IDENTITY, LongString VARCHAR(1000));
INSERT INTO #tbl VALUES('blah')
,('blah blah')
,('blah bleh blih bloh')
,('blah bleh blih bloh bluuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh');
--We can specify the chunk's length
DECLARE #Chunk INT=6;
SELECT t.ID
,A.Nmbr AS ChunkNmbr
,SUBSTRING(t.LongString,A.Nmbr*#Chunk+1,#Chunk) AS ChunkOfString
FROM #tbl t
CROSS APPLY(SELECT TOP(LEN(t.LongString)/#Chunk + 1)
ROW_NUMBER() OVER(ORDER BY (SELECT NULL))-1
FROM master..spt_values) A(Nmbr);
The idea in short:
We use a trick with APPLY and a computed TOP-clause. The source master..spt_values is just a common table with a lot of rows. We don't need the values, just a set to compute a running number using ROW_NUMBER(). APPLY will be called row-by-row. That means, that a long string will create more numbers than a short one.
To get your chunks I use a simple SUBSTRING(), where we compute the start of each chunk by rather simple multiplication.
UPDATE: More than one column in one go
Try this to use this approach for more than one column
DECLARE #tbl TABLE(ID INT IDENTITY, LongString1 VARCHAR(1000), LongString2 VARCHAR(1000));
INSERT INTO #tbl VALUES('blah','dsfafadafdsafdsafdsafsadfdsafdsafdsf')
,('blah blah','afdsafdsafd')
,('blah bleh blih bloh','adfdsafdsafdfdsafdsafdafdsaasdfdsafdsafdsafdsafdsafsadfsadfdsafdsafdsafdsafdafdsafdsafadf')
,('blah bleh blih bloh bluuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh','asdfdsaf');
DECLARE #Chunk INT=6;
SELECT t.ID
,A.MaxLen
,B.Nmbr AS ChunkNmbr
,SUBSTRING(t.LongString1,B.Nmbr*#Chunk+1,#Chunk) AS ChunkOfString1
,SUBSTRING(t.LongString2,B.Nmbr*#Chunk+1,#Chunk) AS ChunkOfString1
FROM #tbl t
CROSS APPLY(SELECT MAX(strLen) FROM (VALUES(LEN(t.LongString1)),(LEN(t.LongString2))) vals(strLen)) A(MaxLen)
CROSS APPLY(SELECT TOP(A.MaxLen/#Chunk + 1)
ROW_NUMBER() OVER(ORDER BY (SELECT NULL))-1
FROM master..spt_values) B(Nmbr);
The new idea: We use an APPLY first to find the longest string in one row. We have to do the chunk computations only for this maximum number.
I have a Redshift table with the following column
How can I extract the value starting by cat_ from this column please (there is only one for each row and at different position in the array)?
I want to get those results:
cat_incident
cat_feature_missing
cat_duplicated_request
Thanks!
There is no easy way to extract multiple values from within one column in SQL (or at least not in the SQL used by Redshift).
You could write a User-Defined Function (UDF) that returns a string containing those values, separated by newlines. Whether this is acceptable depends on what you wish to do with the output (eg JOIN against it).
Another option is to pre-process the data before it is loaded into Redshift, to put this information in a separate one-to-many table, with each value in its own row. It would then be trivial to return this information.
You can do this using tally table (table with numbers). Check this link on information how to create this table: http://www.sqlservercentral.com/articles/T-SQL/62867/
Here is example how you would use it. In real life you should replace temporary #tally table with a permanent one.
--create sample table with data
create table #a (tags varchar(500));
insert into #a
select 'blah,cat_incident,mcr_close_ticket'
union
select 'blah-blah,cat_feature_missing,cat_duplicated_request';
--create tally table
create table #tally(n int);
insert into #tally
select 1
union select 2
union select 3
union select 4
union select 5
;
--get tags
select * from
(
select TRIM(SPLIT_PART(a.tags, ',', t.n)) AS single_tag
from #tally t
inner join #a a ON t.n <= REGEXP_COUNT(a.tags, ',') + 1 and n<1000
)
where single_tag like 'cat%'
;
Thanks!
In the end I managed to do it with the following query:
SELECT SUBSTRING(SUBSTRING(tags, charindex('cat_', tags), len(tags)), 0, charindex(',', SUBSTRING(tags, charindex('cat_', tags), len(tags)))) tags
FROM table
I have two sets and for each value in the first set I want to apply a number of random values from the second. The approach I have chosen uses a select from the first with a cross apply from the second. A simplified MWE is as follows:
DROP TABLE IF EXISTS #S;
CREATE TABLE #S (c CHAR(1));
INSERT INTO #S VALUES ('A'), ('B');
DROP TABLE IF EXISTS #T;
WITH idGen(id) AS (
SELECT 1
UNION ALL
SELECT id + 1 FROM idGen WHERE id < 1000
)
SELECT id INTO #T FROM idGen OPTION(MAXRECURSION 0);
DROP TABLE IF EXISTS #R;
SELECT c, id INTO #R FROM #S
CROSS APPLY (
SELECT id, ROW_NUMBER() OVER (
/*
-- this gives 100% overlap
PARTITION BY c
ORDER BY RAND(CHECKSUM(NEWID()))
*/
-- this gives the expected ~10% overlap
ORDER BY RAND(CHECKSUM(NEWID()) + CHECKSUM(c))
) AS R
FROM #T
) t
WHERE t.R <= 100;
SELECT COUNT(*) AS PercentOverlap -- ~10%
FROM #R rA JOIN #R rB
ON rB.id = rA.id AND rB.c = 'B'
WHERE rA.c = 'A';
While this solution works, I am wondering why changing to the (commented) partitioning method does not? Also, are there any caveats using this solution, seeing as it feels sort of dirty to add two checksums?
In the actual problem there is also a count in the first set containing the number of random values to select from the second set, which replaces the static 100 in the example above. However, using the fixed 100 made it easy to verify the expected overlap.
RAND() function is a run-time constant in SQL Server. It means that usually it is evaluated once for the query. When you pass a value to RAND this value serves as a starting seed.
You need to examine execution plan and you'll see where optimiser puts evaluation of the functions. It the case which doesn't produce expected result most likely optimiser has optimised it too aggressively and moved all "randomness" outside the loop.
Also, there is no point wrapping NEWID() into CHECKSUM() and into RAND().
Simple NEWID() is enough. Or, even better, a function that is designed to produce a random number, such as CRYPT_GEN_RANDOM()
Either version of your query looks a bit strange. I'd write it like this:
SELECT c, id INTO #R
FROM #S
CROSS APPLY
(
SELECT TOP(100) -- or #S.SomeField instead of 100
id
FROM #T
ORDER BY CRYPT_GEN_RANDOM(4) -- generate 4 random bytes, usually it is enough
) AS t
;
This gives 100 random rows from #T for each row from #S.
Actually, the query above is not good. Optimiser again sees that inner query (inside the CROSS APPLY) doesn't depend on outer query and optimises it away.
End result is that random rows are selected only once.
We need something to make optimiser run the inner query for each row from #S.
One way would be something like this:
SELECT c, id INTO #R
FROM #S
CROSS APPLY
(
SELECT TOP(100) -- or #S.SomeField instead of 100
id
FROM #T
ORDER BY CRYPT_GEN_RANDOM(4) + CHECKSUM(c)
) AS t
;
Something in the inner query to reference the row from the outer query. If you put TOP(#S.SomeField) instead of constant TOP(100), then + CHECKSUM(c) is not needed.
This is the plan for the first variant. You can see that #T is scanned once (1000 rows are read).
This is the plan for the second variant. You can see that #T is scanned twice (2000 rows are read).
I want to add a column to my table with a random number using seed.
If I use RAND:
select *, RAND(5) as random_id from myTable
I get an equal value(0.943597390424144 for example) for all the rows, in the random_id column. I want this value to be different for every row - and that for every time I will pass it 0.5 value(for example), it would be the same values again(as seed should work...).
How can I do this?
(
For example, in PostrgreSql I can write
SELECT setseed(0.5);
SELECT t.* , random() as random_id
FROM myTable t
And I will get different values in each row.
)
Edit:
After I saw the comments here, I have managed to work this out somehow - but it's not efficient at all.
If someone has an idea how to improve it - it will be great. If not - I will have to find another way.
I used the basic idea of the example in here.
Creating a temporary table with blank seed value:
select * into t_myTable from (
select t.*, -1.00000000000000000 as seed
from myTable t
) as temp
Adding a random number for each seed value, one row at a time(this is the bad part...):
USE CPatterns;
GO
DECLARE #seed float;
DECLARE #id int;
DECLARE VIEW_CURSOR CURSOR FOR
select id
from t_myTable t;
OPEN VIEW_CURSOR;
FETCH NEXT FROM VIEW_CURSOR
into #id;
set #seed = RAND(5);
WHILE ##FETCH_STATUS = 0
BEGIN
set #seed = RAND();
update t_myTable set seed = #seed where id = #id
FETCH NEXT FROM VIEW_CURSOR
into #id;
END;
CLOSE VIEW_CURSOR;
DEALLOCATE VIEW_CURSOR;
GO
Creating the view using the seed value and ordering by it
create view my_view AS
select row_number() OVER (ORDER BY seed, id) AS source_id ,t.*
from t_myTable t
I think the simplest way to get a repeatable random id in a table is to use row_number() or a fixed id on each row. Let me assume that you have a column called id with a different value on each row.
The idea is just to use this as a seed:
select rand(id*1), as random_id
from mytable;
Note that the seed for the id is an integer and not a floating point number. If you wanted a floating point seed, you could do something with checksum():
select rand(checksum(id*0.5)) as random_id
. . .
If you are doing this for sampling (where you will say random_id < 0.1 for a 10% sample for instance, then I often use modulo arithmetic on row_number():
with t as (
select t.* row_number() over (order by id) as seqnum
from mytable t
)
select *
from t
where ((seqnum * 17 + 71) % 101) < 0.1
This returns about 10% of the numbers (okay, really 10/101). And you can adjust the sample by fiddling with the constants.
Someone sugested a similar query using newid() but I'm giving you the solution that works for me.
There's a workaround that involves newid() instead of rand, but it gives you the same result. You can execute it individually or as a column in a column. It will result in a random value per row rather than the same value for every row in the select statement.
If you need a random number from 0 - N, just change 100 for the desired number.
SELECT TOP 10 [Flag forca]
,1+ABS(CHECKSUM(NEWID())) % 100 AS RANDOM_NEWID
,RAND() AS RANDOM_RAND
FROM PAGSEGURO_WORK.dbo.jobSTM248_tmp_leitores_iso
So, in case it would someone someday, here's what I eventually did.
I'm generating the random seeded values in the server side(Java in my case), and then create a table with two columns: the id and the generated random_id.
Now I create the view as an inner join between the table and the original data.
The generated SQL looks something like that:
CREATE TABLE SEED_DATA(source_id INT PRIMARY KEY, random_id float NOT NULL);
select Rand(5);
insert into SEED_DATA values(1,Rand());
insert into SEED_DATA values(2, Rand());
insert into SEED_DATA values(3, Rand());
.
.
.
insert into SEED_DATA values(1000000, Rand());
and
CREATE VIEW DATA_VIEW
as
SELECT row_number() OVER (ORDER BY random_id, id) AS source_id,column1,column2,...
FROM
( select * from SEED_DATA tmp
inner join my_table i on tmp.source_id = i.id) TEMP
In addition, I create the random numbers in batches, 10,000 or so in each batch(may be higher), so it will not weigh heavily on the server side, and for each batch I insert it to the table in a separate execution.
All of that because I couldn't find a good way to do what I want purely in SQL. Updating row after row is really not efficient.
My own conclusion from this story is that SQL Server is sometimes really annoying...
You could convert a random number from the seed:
rand(row_number over (order by ___, ___,___))
Then cast that as a varchar
, Then use the last 3 characters as another seed.
That would give you a nice random value:
rand(right(cast(rand(row_number() over(x,y,x)) as varchar(15)), 3)
The intention of following (simplified) code fragment is to return one random row.
Unfortunatly, when we run this fragment in the query analyzer, it returns between zero and three results.
As our input table consists of exactly 5 rows with unique ID's and as we perform a select on this table where ID equals a random number, we are stumped that there would ever be more than one row returned.
Note: among other things, we already tried casting the checksum result to an integer with no avail.
DECLARE #Table TABLE (
ID INTEGER IDENTITY (1, 1)
, FK1 INTEGER
)
INSERT INTO #Table
SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
SELECT *
FROM #Table
WHERE ID = ABS(CHECKSUM(NEWID())) % 5 + 1
Edit
Our usage scenario is as follows (please don't comment on wether it is the right thing to do or not. It's the powers that be that have decided)
Ultimately, we must create a result with realistic values where the combination of producer and weights are obfuscated by selecting at random existing weights from the table itself.
The query then would become something like this (also a reason why RAND can not be used)
SELECT t.ID
, FK1 = (SELECT FK1 FROM #Table WHERE ID=ABS(CHECKSUM(NEWID())) % 5 + 1)
FROM #Table t
Because the inner select could be returning zero results, it would return a NULL value wich again is not acceptable. It is the investigation of why the inner select returns between zero and x results, that this question sproused (is this even English?).
Answer
What turned the light on for me was the simple observation that ABS(CHECKSUM(NEWID())) % 5 + 1) was re-evaluated for each row. I was under the impression that ABS(CHECKSUM(NEWID())) % 5 + 1) would get evaluated once, then matched.
Thank you all for answering and slowly but surely leading me to a better understanding.
The reason this happens is because NEWID() gies a different value for each row in the table. For each row, independently of the others, there is a one in five chance of it being returned. Consequently, as it stands, you actually have a 1 in 3125 chance of all 5 rows being returned!
To see this, run the following query. You'll see that each row gets a different ID.
SELECT * , NEWID()
FROM #Table
This will fix your code:
DECLARE #Id int
SET #Id = ABS(CHECKSUM(NEWID())) % 5 + 1
SELECT *
FROM #Table
WHERE ID = #Id
However, I'm not sure this is the most efficient method of selecting a single random row from the table.
You might find this MSDN article useful: http://msdn.microsoft.com/en-us/library/Aa175776 (Random Sampling in T-SQL)
EDIT 1: now I think about it, this probably is the most efficient way to do it, assuming the number of rows remains fixed and the IDs are guaranteed to be contiguous.
EDIT 2: to achieve the desired result when used as a sub-query, use TOP 1 like this:
SELECT t.ID
, FK1 = (SELECT TOP 1 FK1 FROM #Table ORDER BY NEWID())
FROM #Table t
A bit of a guess, and not sure that SQL works this way, but wouldn't SQL evaluate "ABS(CHECKSUM(NEWID())) % 5 + 1" for each row in the table? If so, then each evaluation may or may not return the value of ID of the current row.
Try this instead - generating the random number explicitly first, and matching on that single value:
declare #targetRandom int
set #targetRandom = ABS(CHECKSUM(NEWID())) % 5 + 1
select * from #table where ID = #targetRandom
Try the following, so you can see what happens:
SELECT ABS(CHECKSUM(NEWID())) % 5 + 1 AS Number, #Table.*
FROM #Table
WHERE ID = Number
Or you could use RAND() instead of NEWID(), which is only evaluated once per query in MS SQL
If you want to use CHECKSUM to obtain a random row, this is the way to do it.
SELECT TOP 1 *
FROM #Table
ORDER BY CHECKSUM(NEWID())
what about?
SELECT t.ID
, FK1 = (SELECT TOP 1 FK1 FROM #Table ORDER BY NEWID())
FROM #Table t
This may help you understand the reasons.
Run the query multiple times. How many times does MY_FILTER = ID ?
SELECT *, ABS(CHECKSUM(NEWID())) % 5 + 1 AS MY_FILTER
FROM #Table
SELECT *, ABS(CHECKSUM(NEWID())) % 5 + 1 AS MY_FILTER
FROM #Table
SELECT *, ABS(CHECKSUM(NEWID())) % 5 + 1 AS MY_FILTER
FROM #Table
I don't know how much this will be helpful to you, but try this.. All I understood is you want one random row each time you execute the query..
select top 1 newid() as row,ID from #Table order by row
Here is the logic. Each time you execute the query a newid is being assigned to each row and all are unique and the you just order them with the new uniquely generated rowid. Then all you need to do is select the top most or whatever you want..