Is there a function in PostgreSQL that counts string match across columns (row-wise) - sql

I want to overwrite a number based on a few conditions.
Intended overwrite:
If a string (in the example I use is just a letter) occurs across 3 columns at least 2 times and the numerical column is more than some number, overwrite the numerical value OR
If another string occurs across 3 columns at least 2 times and the numerical column is more than some other number, overwrite the numerical value, else leave the numerical value unchanged.
The approach I thought of first, works but only if the table has one row. Could this be extended somehow so it could work on more rows? And if my approach is wrong, would you please direct me to the right one?
Please, see the SQL Fiddle
Any help is highly appreciated!
if letter a repeats at least 2 times among section_1,section_2,section_3 and number >= 3 then overwrite number with 3 or if letter b repeats at least 2 times among section_1,section_2,section_3 and number >= 8 write 8, else leave number unchanged
CREATE TABLE sections (
id int,
section_1 text,
section_2 text,
section_3 text,
number int
);
INSERT INTO sections VALUES
( 1, 'a', 'a', 'c', 5),
( 2, 'b', 'b', 'c', 9),
( 3, 'b', 'b', 'c', 4);
expected result:
id number
1 3
2 8
3 4


Are you looking for a case expression?
select (case when (section_1 = 'a')::int + (section_2 = 'a')::int + (section_3 = 'a')::int >= 2 and
other_col > threshold
then 'special'
end)
You can have additional when conditions. And include this in an update if you really wand to change the value.


A typical solution uses a lateral join to unpivot:
select s.*, x.number as new_number
from sections s
cross join lateral (
select count(*) number
from (values (s.section_1), (s.section_2), (s.section_3)) x(section)
where section = 'a'
) x;
This is a bit more scalable than repeating conditional expression, since you just need to enumerate the columns in the values() row constructor of the subquery.

Related

Translating an Excel concept into SQL

Let's say I have the following range in Excel named MyRange:
This isn't a table by any means, it's more a collection of Variant values entered into cells. Excel makes it easy to sum these values doing =SUM(B3:D6) which gives 25. Let's not go into the details of type checking or anything like that and just figure that sum will easily skip values that don't make sense.
If we were translating this concept into SQL, what would be the most natural way to do this? The few approaches that came to mind are (ignore type errors for now):
MyRange returns an array of values:
-- myRangeAsList = [1,1,1,2, ...]
SELECT SUM(elem) FROM UNNEST(myRangeAsList) AS r (elem);
MyRange returns a table-valued function of a single column (basically the opposite of a list):
-- myRangeAsCol = (SELECT 1 UNION ALL SELECT 1 UNION ALL ...
SELECT SUM(elem) FROM myRangeAsCol as r (elem);
Or, perhaps more 'correctly', return a 3-columned table such as:
-- myRangeAsTable = (SELECT 1,1,1 UNION ALL SELECT 2,'other',2 UNION ALL ...
SELECT SUM(a+b+c) FROM SELECT a FROM myRangeAsTable (a,b,c)
Unfortunately, I think this makes things the most difficult to work with, as we now have to combine an unknown number of columns.
Perhaps returning a single column is the easiest of the above to work with, but even that takes a very simple concept -- SUM(myRange) and converts into something that is anything but that: SELECT SUM(elem) FROM myRangeAsCol as r (elem).
Perhaps this could also just be rewritten to a function for convenience, for example:

Just possible direction to think
create temp function extract_values (input string)
returns array<string> language js as """
return Object.values(JSON.parse(input));
""";
with myrangeastable as (
select '1' a, '1' b, '1' c union all
select '2', 'other', '2' union all
select 'true', '3', '3' union all
select '4', '4', '4'
)
select sum(safe_cast(value as float64)) range_sum
from myrangeastable t,
unnest(extract_values(to_json_string(t))) value
with output
Note: no columns explicitly used so should work for any sized range w/o any changes in code
Depends on specific use case, I think above can be wrapped into something more friendly for someone who knows excel to do

I'll try to pose, atomic, pure SQL principles that start with obvious items and goes to the more complicated ones. The intention is, all items can be used in any RDBS:
SQL is basically designed to query tabular data which has relations. (Hence the name is Structured Query Language).
The range in excel is a table for SQL. (Yes you can have some other types in different DBs, but keep it simple so you can use the concept in different types of DBs.)
Now we accept a range in the excel is a table in a database. Then the next step is how to map columns and rows of an excel range to a DB table. It is straight forward. An excel range column is a column in DB. And a row is a row. So why is this a separate item? Because the main difference between the two is usually in DBs, adding new column is usually a pain, the DB tables are almost exclusively designed for new rows not for new columns. (But, of course there are methods to add new columns, and even there exists column based DBs, but these are out of the scope of this answer.)
Items 2 and 3 in Excel and in a DB:
/*
Item 2: Table
the range in the excel is modeled as the below test_table
Item 3: Columns
id keeps the excel row number
b, c, d are the corresponding b, c, d columns of the excel
*/
create table test_table
(
id integer,
b varchar(20),
c varchar(20),
d varchar(20)
);
-- Item 3: Adding the rows in the DB
insert into test_table values (3 /* same as excel row number */ , '1', '1', '1');
insert into test_table values (4 /* same as excel row number */ , '2', 'other', '2');
insert into test_table values (5 /* same as excel row number */ , 'TRUE', '3', '3');
insert into test_table values (6 /* same as excel row number */ , '4', '4', '4');
Now we have similar structure. Then the first thing we want to do is to have equal number of rows between excel range and db table. At DB side this is called filtering and your tool is the where condition. where condition goes through all rows (or indexes for the sake of speed but this is beyond this answer's scope), and filters out which does not satisfy the test boolean logic in the condition. (So for example where 1 = 1 is brings all rows because the condition is always true for all rows.
The next thing to do is to sum the related columns. For this purpose you have two options. To use sum(column_a + column_b) (row by row summation) or sum(a) + sum(b) (column by column summation). If we assume all the data are not null, then both gives the same output.
Items 4 and 5 in Excel and in a DB:
select sum(b + c + d) -- Item 5, first option: We sum row by row
from test_table
where id between 3 and 6; -- Item 4: We simple get all rows, because for all rows above the id are between 3 and 6, if we had another row with 7, it would be filtered out
+----------------+
| sum(b + c + d) |
+----------------+
| 25 |
+----------------+
select sum(b) + sum(c) + sum(d) -- Item 5, second option: We sum column by column
from test_table
where id between 3 and 6; -- Item 4: We simple get all rows, because for all rows above the id are between 3 and 6, if we had another row with 7, it would be filtered out
+--------------------------+
| sum(b) + sum(c) + sum(d) |
+--------------------------+
| 25 |
+--------------------------+
At this point it is better to go one step further. In the excel you have got the "pivot table" structure. The corresponding structure at SQL is the powerful group by mechanics. The group by basically groups a table according to its condition and each group behaves like a sub-table. For example if you say group by column_a for a table, the values are grouped according to the values of the table.
SQL is so powerful that you can even filter the sub groups using having clauses, which acts same as where but works over the columns in group by or the functions over those columns.
Items 6 and 7 in Excel and in a DB:
-- Item 6: We can have group by clause to simulate a pivot table
insert into test_table values (7 /* same as excel row */ , '4', '2', '2');
select b, sum(d), min(d), max(d), avg(d)
from test_table
where id between 3 and 7
group by b;
+------+--------+--------+--------+--------+
| b | sum(d) | min(d) | max(d) | avg(d) |
+------+--------+--------+--------+--------+
| 1 | 1 | 1 | 1 | 1 |
| 2 | 2 | 2 | 2 | 2 |
| TRUE | 3 | 3 | 3 | 3 |
| 4 | 6 | 2 | 4 | 3 |
+------+--------+--------+--------+--------+
Beyond this point following are the details which are not directly related with the questions purpose:
SQL has the ability for table joins (the relations). They can be thought like the VLOOKUP functionality in the Excel.
The RDBSs have the indexing mechanisms to fetch the rows as quick as possible. (Where the RDBMSs start to go beyond the purpose of excel).
The RDBSs keep huge amount of data (where excel the max rows are limited).
Both RDBSMs and Excel can be used by most of frameworks as persistent data layer. But of course Excel is not the one you pick because its reason of existence is more on the presentation layer.
The excel file and the SQL used in this answer can be found in this github repo: https://github.com/MehmetKaplan/stackoverflow-72135212/
PS: I used SQL for more than 2 decades and then reduced using it and started to use Excel much frequently because of job changes. Each time I use Excel I still think of the DBs and "relational algebra" which is the mathematical foundation of the RDBMSs.

So in Snowflake:
Strings as input:
if you have your data in a "order" table represented by this CTE:
and the data was strings of comma separated values:
WITH data(raw) as (
select * from values
('null,null,null,null,null,null'),
('null,null,null,null,null,null'),
('null,1,1,1,null,null'),
('null,2, other,2,null,null'),
('null,true,3,3,null,null'),
('null,4,4,4,null,null')
)
this SQL will select the sub part, try parse it and sum the valid values:
select sum(nvl(try_to_double(r.value::text), try_to_number(r.value::text))) as sum_total
from data as d
,table(split_to_table(d.raw,',')) r
where r.index between 2 and 4 /* the column B,C,D filter */
and r.seq between 3 and 6 /* the row 3-6 filter */
;
giving:
SUM_TOTAL
25
Arrays as input:
if you already have arrays.. here I am smash those strings into STRTOK_TO_ARRAY in the CTE to make me some arrays:
WITH data(_array) as (
select STRTOK_TO_ARRAY(column1, ',') from values
('null,null,null,null,null,null'),
('null,null,null,null,null,null'),
('null,1,1,1,null,null'),
('null,2, other,2,null,null'),
('null,true,3,3,null,null'),
('null,4,4,4,null,null')
)
thus again with almost the same SQL, but not the array indexes are 0 based, and I have used FLATTEN:
select sum(nvl(try_to_double(r.value::text), try_to_number(r.value::text))) as sum_total
from data as d
,table(flatten(input=>d._array)) r
where r.index between 1 and 3 /* the column B,C,D filter */
and r.seq between 3 and 6 /* the row 3-6 filter */
;
gives:
SUM_TOTAL
25
With JSON driven data:
This time using semi-structured data, we can include the filter ranges with the data.. and some extra "out of bounds values just to show we are not just converting it all.
WITH data as (
select parse_json('{ "col_from":2,
"col_to":4,
"row_from":3,
"row_to":6,
"data":[[101,102,null,104,null,null],
[null,null,null,null,null,null],
[null,1,1,1,null,null],
[null,2, "other",2,null,null],
[null,true,3,3,null,null],
[null,4,4,4,null,null]
]}') as json
)
select
sum(try_to_double(c.value::text)) as sum_total
from data as d
,table(flatten(input=>d.json:data)) r
,table(flatten(input=>r.value)) c
where r.index+1 between d.json:row_from::number and d.json:row_to::number
and c.index+1 between d.json:col_from::number and d.json:col_to::number
;

Here is another solution using Snowflake scripting (Snowsight format) . This code can easily be wrapped as a stored procedure.
declare
table_name := 'xl_concept'; -- input
column_list := 'a,b,c'; -- input
total resultset; -- result output
pos int := 0; -- position for delimiter
sql := ''; -- sql to be generated
col := ''; -- individual column names
begin
sql := 'select sum('; -- initialize sql
loop -- repeat until column list is empty
col := replace(split_part(:column_list, ',', 1), ',', ''); -- get the column name
pos := position(',' in :column_list); -- find the delimiter
sql := sql || 'coalesce(try_to_number('|| col ||'),0)'; -- add to the sql
if (pos > 0) then -- more columns in the column list
sql := sql || ' + ';
column_list := right(:column_list, len(:column_list) - :pos); -- update column list
else -- last entry in the columns list
break;
end if;
end loop;
sql := sql || ') total from ' || table_name||';'; -- finalize the sql
total := (execute immediate :sql); -- run the sql and store total value
return table(total); -- return total value
end;
only these two variables need to be set table_name and column_list
generates the following sql to sum up the values
select sum(coalesce(try_to_number(a),0) + coalesce(try_to_number(b),0) + coalesce(try_to_number(c),0)) from xl_concept
prep steps
create or replace temp table xl_concept (a varchar,b varchar,c varchar)
;
insert into xl_concept
with cte as (
select '1' a, '1' b, '1' c union all
select '2', 'other', '2' union all
select 'true', '3', '3' union all
select '4', '4', '4'
)
select * from cte
;
result for the run with no change
TOTAL
25
result after changing column list to column_list := 'a,c';
TOTAL
17
Also, this can be enhanced setting columns_list to * and reading the column names from information_schema.columns to include all the columns from the table.

In PostgreSQL regular expression can be used to filter non numeric values before sum
select sum(e::Numeric) from (
select e
from unnest((Array[['1','2w','1.2e+4'],['-1','2.232','zz']])) as t(e)
where e ~ '^[-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?$'
) a
expression for validating numeric value was taken from post Return Just the Numeric Values from a PostgreSQL Database Column
More secure option is to define function as in PostgreSQL alternative to SQL Servers try_cast function
Function (simplified for this example):
create function try_cast_numeric(p_in text)
returns Numeric
as
$$
begin
begin
return $1::Numeric;
exception
when others then
return 0;
end;
end;
$$
language plpgsql;
Select
select
sum(try_cast_numeric(e))
from
unnest((Array[['1','2w','1.2e+4'],['-1','2.232','zz']])) as t(e)

Most modern RDBMS support lateral joins and table value constructors. You can use them together to convert arbitrary columns to rows (3 columns per row become 3 rows with 1 column) then sum. In SQL server you would:
create table t (
id int not null primary key identity,
a int,
b int,
c int
);
insert into t(a, b, c) values
( 1, 1, 1),
( 2, null, 2),
(null, 3, 3),
( 4, 4, 4);
select sum(value)
from t
cross apply (values
(a),
(b),
(c)
) as x(value);
Below is the implementation of this concept in some popular RDBMS:
SQL Server
PostgreSQL
MySQL
Generic solution, ANSI SQL
Unpivot solution, Oracle

Using regular expression to extract all number values from a row could be another option, I guess.
DECLARE rectangular_table ARRAY<STRUCT<A STRING, B STRING, C STRING>> DEFAULT [
('1', '1', '1'), ('2', 'other', '2'), ('TRUE', '3', '3'), ('4', '4', '4')
];
SELECT SUM(SAFE_CAST(v AS FLOAT64)) AS `sum`
FROM UNNEST(rectangular_table) t,
UNNEST(REGEXP_EXTRACT_ALL(TO_JSON_STRING(t), r':"?([-0-9.]*)"?[,}]')) v
output:
+------+------+
| Row | sum |
+------+------+
| 1 | 25.0 |
+------+------+

You could use a CTE with a SELECT FROM VALUES
with xlary as
(
select val from (values
('1')
,('1')
,('1')
,('2')
,('OTHER')
,('2')
,('TRUE')
,('3')
,('3')
,('4')
,('4')
,('4')
) as tbl (val)
)
select sum(try_cast(val as number)) from xlary;

Given a table of numbers, can I get all the rows which add up to less than or equal to a number?

Say I have a table with an incrementing id column and a random positive non zero number.
id
rand
1
12
2
5
3
99
4
87
Write a query to return the rows which add up to a given number.
A couple rules:
Rows must be "consumed" in order, even if a later row makes it a a perfect match. For example, querying for 104 would be a perfect match for rows 1, 2, and 4 but rows 1-3 would still be returned.
You can use a row partially if there is more available than is necessary to add up to whatever is leftover on the number E.g. rows 1, 2, and 3 would be returned if your max number is 50 because 12 + 5 + 33 equals 50 and 90 is a partial result.
If there are not enough rows to satisfy the amount, then return ALL the rows. E.g. in the above example a query for 1,000 would return rows 1-4. In other words, the sum of the rows should be less than or equal to the queried number.
It's possible for the answer to be "no this is not possible with SQL alone" and that's fine but I was just curious. This would be a trivial problem with a programming language but I was wondering what SQL provides out of the box to do something as a thought experiment and learning exercise.

You didn't mention which RDBMS, but assuming SQL Server:
DROP TABLE #t;
CREATE TABLE #t (id int, rand int);
INSERT INTO #t (id,rand)
VALUES (1,12),(2,5),(3,99),(4,87);
DECLARE #target int = 104;
WITH dat
AS
(
SELECT id, rand, SUM(rand) OVER (ORDER BY id) as runsum
FROM #t
),
dat2
as
(
SELECT id, rand
, runsum
, COALESCE(LAG(runsum,1) OVER (ORDER BY id),0) as prev_runsum
from dat
)
SELECT id, rand
FROM dat2
WHERE #target >= runsum
OR #target BETWEEN prev_runsum AND runsum;

How can I convert 2 row into column in tsql?

I have 2 row data which I want to make it to be 2 column,
I tried union syntax but it didn't work.
Here is the data I have:
breed 1 breed2
I tried to convert it with this sql
select a.breed union a.breed
but it didn't work.
Here is what you want from the SQL:
breed1,breed2

SELECT
[breed1],
[breed2]]
FROM
(
SELECT 'breed1' myColumn
union
select 'breed2'
) AS SourceTable
PIVOT
(
AVG(mySecondColumn) FOR
myColumn IN ([breed1], [breed2]])
) AS PivotTable;

You can use a self join. This needs a way to pair rows together (so if you have four rows you get 1 and 2 in one result and 3 and 4 in the other rather than another combination).
I'm going to assume you have sequentially numbered rows in an Id column and an odd numbered row is paired with the one greater even Id:
select odd.Data as 'First', even.Data as 'Second'
from TheData odd
inner join TheData even on odd.Id+1 = even.Id
where odd.Id % 2 = 1;
More generally for more columns use of pivot is more flexible.

How about an aggregation query?
select min(breed) as breed1, max(breed) as breed2
from t;

SQL : select one column if two columns are equal, else select both

I have the following SQL select problem:
I have two columns positive threshold and negative threshold (among several other columns like name, ids.... ).
If their (absolute) value is the same (multiply by -1) then I want to select only the positive threshold as column [threshold].
If the values are different, I want to select two columns [positiveThreshold] and [negativeThreshold].
Thank you in advance.

select null as [threshold], positivethreshold, negativethreshold
from table
where negativethreshold is null
or (positivethreshold + negativethreshold) <> 0
union
select positivethreshold, null, null
from table
where (positivethreshold + negativethreshold) = 0

Quicker with SQL to SUM 0 values or exclude them?

I want to SUM a lot of rows.
Is it quicker (or better practice, etc) to do Option A or Option B?
Option A
SELECT
[Person]
SUM([Value]) AS Total
FROM
Database
WHERE
[Value] > 0
GROUP BY
[Person]
Option B
SELECT
[Person]
SUM([Value]) AS Total
FROM
Database
GROUP BY
[Person]
So if I have, for Person X:
0, 7, 0, 6, 0, 5, 0, 0, 0, 4, 0, 9, 0, 0
Option A does:
a) Remove zeros
b) 7 + 6 + 5 + 4 + 9
Option B does:
a) 0 + 7 + 0 + 6 + 0 + 5 + 0 + 0 + 0 + 4 + 0 + 9 + 0 + 0
Option A has less summing, because it has fewer records to sum, because I've excluded the load that have a zero value. But Option B doesn't need a WHERE clause.
Anyone got an idea as to whether either of these are significantly quicker/better than the other? Or is it just something that doesn't matter either way?
Thanks :-)

Well, if you have a filtered index that exactly matches the where clause, and if that index removes a significant amount of data (as in: a good chunk of the data is zeros), then definitely the first... If you don't have such an index: then you'll need to test it on your specific data, but I would probably expect the unfiltered scenario to be faster, as it can do use a range of tricks to do the sum if it doesn't need to do branching etc.
However, the two examples aren't functionally equivalent at the moment (the second includes negative values, the first doesn't).

Assuming that Value is always positive the 2nd query might still return less rows if there's a Person with all zeroes.
Otherwise you should simply test actual runtime/CPU on a really large amount of rows.

As already pointed out, the two are not functionally equivalent. In addition to the differences already pointed out (negative values, different output row count), Option B also filters out rows where Value is NULL. Option A doesn't.

Based on the Execution plan for both of these and using a small dataset similar to the one you provided, Option B is slightly faster with an Estimated Subtree Cost of .0146636 vs .0146655. However, you may get different results depending on the query or size of dataset. Only option is to test and see for yourself.
http://www.developer.com/db/how-to-interpret-query-execution-plan-operators.html
Drop Table #Test
Create Table #Test (Person nvarchar(200), Value int)
Insert Into #Test
Select 'Todd', 12 Union
Select 'Todd', 11 Union
Select 'Peter', 20 Union
Select 'Peter', 29 Union
Select 'Griff', 10 Union
Select 'Griff', 0 Union
Select 'Peter', 0 Union
SELECT [Person], SUM([Value]) AS Total
FROM #Test
WHERE [Value] > 0
GROUP BY [Person]
SELECT [Person],SUM([Value]) AS Total
FROM #Test
GROUP BY [Person]