Coming to BigQuery somewhat recently from Postgres - I'm used to utilizing the following pattern fairly regularly in specific cases, but not where the CTE set would be too big of course.
with cte1 as (
select id from my_table where feature1 = x, feature2 = y
)
select id, detail1, detail2, detail3 from my_table
where id = ANY( select id from cte1 )
Something to the effect of using my CTE as a place to gather records of a particular interest (something in common, something uncommon, outliers, dupes etc.) and then passing that to my main select using the id = ANY( select id from CTE ) pattern.
What would be the equivalent in BigQuery?
Equivalent is
where id in ( select id from CTE )
So, your query can look like below
with cte1 as (
select id from my_table where feature1 = x and feature2 = y
)
select id, detail1, detail2, detail3 from my_table
where id in ( select id from cte1 )
Tables: I have 3 tables
They are cust, new_cust, old_cust
all of them have 3 columns, they are id, username, name
each of them have possibilities to have same data as the others.
I would like to make "whole" table that consisting all of them but only the uniques.
I've Tried
Creating a dummy table
I've tried to create the dummy table called "Temp" table by
select *
into Temp
from cust
insert all table to dummy
Then I insert all of them into they Temp table
insert into temp
select * from new_cust
insert into temp
select * from old_cust
taking uniques using distinct
After they all merged I'm using distinct to only take the unique id value
select distinct(id), username, fullname
into Whole
from temp
it did decreasing some rows
Result
But after I move it to whole table I would like to put primary key on id but I got the message that there are some duplicate values. Is there any other way?
I am guessing that you want unique ids. And you want these prioritized by the tables in some order. If so, you can do this with union all and row_number():
select id, username, name
from (select c.*,
row_number() over (partition by id order by priority) as seqnum
from ((select id, username, name, 1 as priority
from new_cust
) union all
(select id, username, name, 2 as priority
from cust
) union all
(select id, username, name, 3 as priority
from old_cust
)
) c
) c
where seqnum = 1;
Try this:
insert into temp
select * from new_cust
UNION
select * from old_cust
Union will avoid the duplicate entries and you can then create a primary key on ID column
Try this below query...
WITH cte as (
SELECT id, username, NAME,
ROW_NUMBER() OVER (PARTITION BY t1.id ORDER BY t1.username, t1.name ) AS rn
FROM cust t1
LEFT JOIN new_cust t2 ON t1.Id = t2.Id
LEFT JOIN old_cust t3 ON t2.Id = t3.Id
)
SELECT id, username, NAME
FROM cte
WHERE rn = 1
Note:-
Put all the query inside a CTE(Common table expression)
with a new column(rn) that you will use to filter the results.
This new Column will produce ROW_NUMBER()....PARTITION BY username,name.....
But after I move it to whole table I would like to put primary key on
id but I got the message that there are some duplicate values.?
That's because You are trying to insert ID value from each of the tables to Whole table.
Just insert username and name and skip ID. ID is IDENTITY and it MUST be unique.
Run this on Your current Whole table to see if You have duplicated Id's:
select COUNT(ID), username
from whole
GROUP BY username
HAVING COUNT(ID) > 1
To get unique customers recreate table Whole and make ID col IDENTITY:
IF OBJECT_ID ('dbo.Whole') IS NOT NULL DROP TABLE dbo.Whole;
CREATE TABLE Whole (ID INT NOT NULL IDENTITY(1,1), Name varchar(max), Username varchar(max))
Insert values into Whole table:
INSERT INTO Whole
SELECT Name, Username FROM cust
UNION
SELECT Name, Username FROM new_cust
UNION
SELECT Name, Username FROM old_cust
Make ID col PK.
What does Unique mean for your row ?
If it is only the username, and you don't care about keeping the old ID values,
this will favor the new_cust data over the old_cust data.
SELECT
ID = ROW_NUMBER() OVER (ORDER BY all_temp.username)
, all_temp.*
INTO dbo.Temp
FROM
(
SELECT nc.username, nc.[name] FROM new_cust AS nc
UNION
SELECT oc.username, oc.[name]
FROM old_cust AS oc
WHERE oc.username NOT IN (SELECT nc1.username FROM new_cust AS nc1) --remove the where part if needed
) AS all_temp
ALTER TABLE dbo.Temp ALTER COLUMN ID INTEGER NOT NULL
ALTER TABLE dbo.Temp ADD PRIMARY KEY (ID)
If by Unique you mean both the username and name then just remove the where part in the union
I have almost the same problem as described in this question. I have two tables with the same number of rows, and I would like to join them together one by one.
The tables are ordered, and I would like to keep this order after the join, if it is possible.
There is a rowid based solution for MSSql, but in SQLite rowid can not be used if the table is coming from a WITH statement (or RECURSIVE WITH).
It is guaranteed that the two tables have the exact same number of rows, but this number is not known beforehand. It is also important to note, that the same element may occur more than twice. The results are ordered, but none of the columns are unique.
Example code:
WITH
table_a (n) AS (
SELECT 2
UNION ALL
SELECT 4
UNION ALL
SELECT 5
),
table_b (s) AS (
SELECT 'valuex'
UNION ALL
SELECT 'valuey'
UNION ALL
SELECT 'valuez'
)
SELECT table_a.n, table_b.s
FROM table_a
LEFT JOIN table_b ON ( table_a.rowid = table_b.rowid )
The result I would like to achieve is:
(2, 'valuex'),
(4, 'valuey'),
(5, 'valuez')
SQLFiddle: http://sqlfiddle.com/#!5/9eecb7/6888
This is quite complicated in SQLite -- because you are allowing duplicates. But you can do it. Here is the idea:
Summarize the table by the values.
For each value, get the count and offset from the beginning of the values.
Then use a join to associate the values and figure out the overlap.
Finally use a recursive CTE to extract the values that you want.
The following code assumes that n and s are ordered -- as you specify in your question. However, it would work (with small modifications) if another column specified the ordering.
You will notice that I have included duplicates in the sample data:
WITH table_a (n) AS (
SELECT 2 UNION ALL
SELECT 4 UNION ALL
SELECT 4 UNION ALL
SELECT 4 UNION ALL
SELECT 5
),
table_b (s) AS (
SELECT 'valuex' UNION ALL
SELECT 'valuey' UNION ALL
SELECT 'valuey' UNION ALL
SELECT 'valuez' UNION ALL
SELECT 'valuez'
),
a as (
select a.n, count(*) as a_cnt,
(select count(*) from table_a a2 where a2.n < a.n) as a_offset
from table_a a
group by a.n
),
b as (
select b.s, count(*) as b_cnt,
(select count(*) from table_b b2 where b2.s < b.s) as b_offset
from table_b b
group by b.s
),
ab as (
select a.*, b.*,
max(a.a_offset, b.b_offset) as offset,
min(a.a_offset + a.a_cnt, b.b_offset + b.b_cnt) - max(a.a_offset, b.b_offset) as cnt
from a join
b
on a.a_offset + a.a_cnt - 1 >= b.b_offset and
a.a_offset <= b.b_offset + b.b_cnt - 1
),
cte as (
select n, s, offset, cnt, 1 as ind
from ab
union all
select n, s, offset, cnt, ind + 1
from cte
where ind < cnt
)
select n, s
from cte
order by n, s;
Here is a DB Fiddle showing the results.
I should note that this would be much simpler in almost any other database, using window functions (or perhaps variables in MySQL).
Since the tables are ordered, you can add row_id values by comparing n values.
But still the best way in order to get better performance would be inserting the ID values while creating the tables.
http://sqlfiddle.com/#!5/9eecb7/7014
WITH
table_a_a (n, id) AS
(
WITH table_a (n) AS
(
SELECT 2
UNION ALL
SELECT 4
UNION ALL
SELECT 5
)
SELECT table_a.n, (select count(1) from table_a b where b.n <= table_a.n) id
FROM table_a
) ,
table_b_b (n, id) AS
(
WITH table_a (n) AS
(
SELECT 'valuex'
UNION ALL
SELECT 'valuey'
UNION ALL
SELECT 'valuez'
)
SELECT table_a.n, (select count(1) from table_a b where b.n <= table_a.n) id
FROM table_a
)
select table_a_a.n,table_b_b.n from table_a_a,table_b_b where table_a_a.ID = table_b_b.ID
or convert the input set to comma separated list and try like this:
http://sqlfiddle.com/#!5/9eecb7/7337
WITH RECURSIVE table_b( id,element, remainder ) AS (
SELECT 0,NULL AS element, 'valuex,valuey,valuz,valuz' AS remainder
UNION ALL
SELECT id+1,
CASE
WHEN INSTR( remainder, ',' )>0 THEN
SUBSTR( remainder, 0, INSTR( remainder, ',' ) )
ELSE
remainder
END AS element,
CASE
WHEN INSTR( remainder, ',' )>0 THEN
SUBSTR( remainder, INSTR( remainder, ',' )+1 )
ELSE
NULL
END AS remainder
FROM table_b
WHERE remainder IS NOT NULL
),
table_a( id,element, remainder ) AS (
SELECT 0,NULL AS element, '2,4,5,7' AS remainder
UNION ALL
SELECT id+1,
CASE
WHEN INSTR( remainder, ',' )>0 THEN
SUBSTR( remainder, 0, INSTR( remainder, ',' ) )
ELSE
remainder
END AS element,
CASE
WHEN INSTR( remainder, ',' )>0 THEN
SUBSTR( remainder, INSTR( remainder, ',' )+1 )
ELSE
NULL
END AS remainder
FROM table_a
WHERE remainder IS NOT NULL
)
SELECT table_b.element, table_a.element FROM table_b, table_a WHERE table_a.element IS NOT NULL and table_a.id = table_b.id;
SQL
SELECT a1.n, b1.s
FROM table_a a1
LEFT JOIN table_b b1
ON (SELECT COUNT(*) FROM table_a a2 WHERE a2.n <= a1.n) =
(SELECT COUNT(*) FROM table_b b2 WHERE b2.s <= b1.s)
Explanation
The query simply counts the number of rows up until the current one for each table (based on the ordering column) and joins on this value.
Demo
See SQL Fiddle demo.
Assumptions
A single column in used for the ordering in each table. (But the query could easily be modified to allow multiple ordering columns).
The ordering values in each table are unique.
The values in the ordering column aren't necessarily the same between the two tables.
It is known that table_a contains either the same or more rows than table_b. (If this isn't the case then a FULL OUTER JOIN would need to be emulated since SQLite doesn't provide one.)
No further changes to the table structure are allowed. (If they are, it would be more efficient to have pre-populated columns for the ordering).
Either way...
Use something like
WITH
v_table_a (n, rowid) AS (
SELECT 2, 1
UNION ALL
SELECT 4, 2
UNION ALL
SELECT 5, 3
),
v_table_b (s, rowid) AS (
SELECT 'valuex', 1
UNION ALL
SELECT 'valuey', 2
UNION ALL
SELECT 'valuez', 3
)
SELECT v_table_a.n, v_table_b.s
FROM v_table_a
LEFT JOIN v_table_b ON ( v_table_a.rowid = v_table_b.rowid );
for "virtual" tables (with WITH or without),
WITH RECURSIVE vr_table_a (n, rowid) AS (
VALUES (2, 1)
UNION ALL
SELECT n + 2, rowid + 1 FROM vr_table_a WHERE rowid < 3
)
, vr_table_b (s, rowid) AS (
VALUES ('I', 1)
UNION ALL
SELECT s || 'I', rowid + 1 FROM vr_table_b WHERE rowid < 3
)
SELECT vr_table_a.n, vr_table_b.s
FROM vr_table_a
LEFT JOIN vr_table_b ON ( vr_table_a.rowid = vr_table_b.rowid );
for "virtual" tables using recursive WITHs (in this example the values are others then yours, but I guess you get the point) and
CREATE TABLE p_table_a (n INT);
INSERT INTO p_table_a VALUES (2), (4), (5);
CREATE TABLE p_table_b (s VARCHAR(6));
INSERT INTO p_table_b VALUES ('valuex'), ('valuey'), ('valuez');
SELECT p_table_a.n, p_table_b.s
FROM p_table_a
LEFT JOIN p_table_b ON ( p_table_a.rowid = p_table_b.rowid );
for physical tables.
I'd be careful with the last one though. A quick test shows, that the numbers of rowid are a) reused -- when some rows are deleted and others are inserted, the inserted rows get the rowids from the old rows (i.e. rowid in SQLite isn't unique past the lifetime of a row, whereas e.g. Oracle's rowid AFAIR is) -- and b) corresponds to the order of insertion. But I don't know and didn't find a clue in the documentation, if that's guaranteed or is subject to change in other/future implementations. Or maybe it's just a mere coincidence in my test environment.
(In general physical order of rows may be subject to change (even within the same database using the same DMBS as a result of some reorganization) and is therefore no good choice to rely on. And it's not guaranteed, a query will return the result ordered by physical position in the table as well (it might use the order of some index instead or have a partial result ordered some other way influencing the output's order). Consider designing your tables using common (sort) keys in corresponding rows for ordering and to join on.)
You can create temp tables to carry CTE data row. then JOIN them by sqlite row_id column.
CREATE TEMP TABLE temp_a(n integer);
CREATE TEMP TABLE temp_b(n VARCHAR(255));
WITH table_a(n) AS (
SELECT 2 n
UNION ALL
SELECT 4
UNION ALL
SELECT 5
UNION ALL
SELECT 5
)
INSERT INTO temp_a (n) SELECT n FROM table_a;
WITH table_b (n) AS
(
SELECT 'valuex'
UNION ALL
SELECT 'valuey'
UNION ALL
SELECT 'valuez'
UNION ALL
SELECT 'valuew'
)
INSERT INTO temp_b (n) SELECT n FROM table_b;
SELECT *
FROM temp_a a
INNER JOIN temp_b b on a.rowid = b.rowid;
sqlfiddle:http://sqlfiddle.com/#!5/9eecb7/7252
It is possible to use the rowid inside a with statement but you need to select it and make it available to the query using it.
Something like this:
with tablea AS (
select id, rowid AS rid from someids),
tableb AS (
select details, rowid AS rid from somedetails)
select tablea.id, tableb.details
from
tablea
left join tableb on tablea.rid = tableb.rid;
It is however as they have already warned you a really bad idea. What if the app breaks after inserting in one table but before the other one? What if you delete an old row? If you want to join two tables you need to specify the field to do so. There are so many things that could go wrong with this design. The most similar thing to this would be an incremental id field that you would save in the table and use in your application. Even simpler, make those into one table.
Read this link for more information about the rowid: https://www.sqlite.org/lang_createtable.html#rowid
sqlfiddle: http://sqlfiddle.com/#!7/29fd8/1
It is possible to use the rowid inside a with statement but you need to select it and make it available to the query using it. Something like this:
with tablea AS (select id, rowid AS rid from someids),
tableb AS (select details, rowid AS rid from somedetails)
select tablea.id, tableb.details
from
tablea
left join tableb on tablea.rid = tableb.rid;
The problem statement indicates:
The tables are ordered
If this means that the ordering is defined by the ordering of the values in the UNION ALL statements, and if SQLite respects that ordering, then the following solution may be of interest because, apart from small tweaks to the last three lines of the sample program, it adds just two lines:
A(rid,n) AS (SELECT ROW_NUMBER() OVER ( ORDER BY 1 ) rid, n FROM table_a),
B(rid,s) AS (SELECT ROW_NUMBER() OVER ( ORDER BY 1 ) rid, s FROM table_b)
That is, table A is table_a augmented with a rowid, and similarly for table B.
Unfortunately, there is a caveat, though it might just be the result of my not having found the relevant specifications. Before delving into that, however, here is the full proposed solution:
WITH
table_a (n) AS (
SELECT 2
UNION ALL
SELECT 4
UNION ALL
SELECT 5
),
table_b (s) AS (
SELECT 'valuex'
UNION ALL
SELECT 'valuey'
UNION ALL
SELECT 'valuez'
),
A(rid,n) AS (SELECT ROW_NUMBER() OVER ( ORDER BY 1 ) rid, n FROM table_a),
B(rid,s) AS (SELECT ROW_NUMBER() OVER ( ORDER BY 1 ) rid, s FROM table_b)
SELECT A.n, B.s
FROM A LEFT JOIN B
ON ( A.rid = B.rid );
Caveat
The proposed solution has been tested against a variety of data sets using sqlite version 3.29.0, but whether or not it is, and will continue to be, "guaranteed" to work is unclear to me.
Of course, if SQLite offers no guarantees with respect to the ordering of the UNION ALL statements (that is, if the question is based on an incorrect assumption), then it would be interesting to see a well-founded reformulation.
I have a table form which I need to extract some information. This table has an oracle spatial (MDSYS.SDO_GEOMETRY) column, from which I also need some data.
I started out with a simple query like this:
select id, field1, field2
FROM my_table;
After that, I was able to loop over the result to extract the data that was in the spatial column:
SELECT *
FROM TABLE (SELECT a.POSITIONMAP.sdo_ordinates
FROM my_table
WHERE ID = 18742084);
The POSITIONMAP.sdo_ordinates seems to usually hold 4 values, like these:
100050,887
407294,948
0,577464740471056
-0,816415625470689
I need the last 2 values. I can achieve that by changing the query into this:
SELECT * FROM
(SELECT rownum AS num,
column_value AS orientatie
FROM TABLE (SELECT a.POSITIONMAP.sdo_ordinates
FROM my_table
WHERE ID = 18742084))
WHERE num IN (3,4)
Looping over every row from my first query to extract the data from the POSITIONMAP column is of course not very performance friendly, so my query becomes slow very quickly.
I would like to retrieve all information in one query, but there are a few things that prevent me from doing so.
Not every row in the table has data in POSITIONMAP
Some rows do have data in POSITIONMAP, but they only contain 2 values (so not the 3rd and 4th value that I am looking for.
I need the data in one row for every row in the table (using the previous query would result in duplicate rows
The closest I got is:
select
id,
field1,
field2
t.*
FROM my_table v,
table (v.POSITIONMAP.sdo_ordinates) t
This gives my 4 rows for every row in my_table.
As soon as I try to put the rownum condition into this query, I get an error: "invalid user.table.column, table.column, or column specification"
Is there any way to combine what I want to do into 1 query?
You can use sdo_util.getvertices as follows:
select t.x,t.y
from my_table mt
,table(sdo_util.getvertices(mt.positionmap)) t
where t.id = 2
I'm assuming that your geometries are lines (gtype=2002) and points (gtype= 2001). If you want X,Y values for lines and empty values for point you can filter on the sdo_gtype property of the geometry object.
select t.x,t.y
from my_table mt
,table(sdo_util.getvertices(mt.positionmap)) t
where t.id = 2
and mt.positionmap.sdo_gtype=2002
union all
select null as X,
null as Y
from my_table mt
where mt.positionmap.sdo_gtype=2001
One method is to use the ROW_NUMBER() analytic function:
SELECT *
FROM (
select id,
field1,
field2,
t.*,
ROW_NUMBER() OVER ( PARTITION BY v.id ORDER BY ROWNUM ) AS rn
FROM my_table v,
TABLE( v.POSITIONMAP.sdo_ordinates ) t
)
WHERE rn IN ( 3, 4 )
Lets assume my data looks like this :
Every second row represents old (previous value) in a table that holds historical data.
table 1 :
id value
------------
1 a
1 b
2 c
2 d
3 a
3 b
and i want to get value of every second row to appear in new 3rd column like this :
table 2:
id new_value old_value
------------------------
1 a b
2 c d
3 a b
EDIT:
For clarity ill post the skeleton of query thats producing data i want to transform (so its clear i am already using WITH so cant use additional one due to oracle not yet allowing nesting of WITH elements) :
skeleton code that produces data in table 1 :
with candidates as
(
--select list of candidates
)
SELECT * FROM
(
(
--select new values
MINUS
--select old values
)
UNION
(
--select old values
MINUS
--select new values
)
)
ORDER BY id;
The goal is to finally get only a list of ids that changed with their old and new values.
Thanks in advance.
Use CTE
;WITH CTE AS(
SELECT *, ROW_NUMBER() OVER (PARTITION BY ID ORDER BY ID) RN
FROM TableName
)
SELECT ID,
MIN(CASE WHEN RN=1 THEN [value] END) NewValue,
MIN(CASE WHEN RN=2 THEN [value] END) OldValue
FROM CTE
GROUP BY ID
It is quite possible that overall query can be written in a much simpler way. Just join intermediary results with old and new values together on id to put them in two different columns instead of unioning them into the same column.
WITH
candidates
AS
(
--select list of candidates
)
,CTE_NewValues
AS
(
--select new values
select id, value AS new_value
FROM candidates
WHERE ...
-- assumes id is unique, one row per id
)
,CTE_OldValues
AS
(
--select old values
select id, value AS old_value
FROM candidates
WHERE ...
-- assumes id is unique, one row per id
)
SELECT
CTE_NewValues.id
,CTE_NewValues.new_value
,CTE_OldValues.old_value
FROM
CTE_NewValues
INNER JOIN CTE_OldValues ON CTE_NewValues.id = CTE_OldValues.id
WHERE
CTE_NewValues.new_value <> CTE_OldValues.old_value
ORDER BY
CTE_NewValues.id;
If we stick to the skeleton of the query in the question, there are also many ways to do it. Self-join is likely to be less efficient than using analytic functions, like ROW_NUMBER and LEAD.
Sorting just by id is not enough to unambiguously define which value is new or old. You need to have some extra column to resolve it.
You don't "nest" WITH (common-table expressions), you "chain" them. Something like the following. As you do that, make sure to add the sort_order column to be able to distinguish old and new values, if you don't have a similar column already.
WITH
candidates
AS
(
--select list of candidates
)
,CTE_YourQuery
AS
(
SELECT * FROM
(
(
--select new values
select 1 AS sort_order, id, value
MINUS
--select old values
select 1 AS sort_order, id, value
)
UNION ALL
(
--select old values
select 2 AS sort_order, id, value
MINUS
--select new values
select 2 AS sort_order, id, value
)
)
)
,CTE_RowNumber
AS
(
SELECT
id
,value AS new_value
,ROW_NUMBER() OVER (PARTITION BY id ORDER BY sort_order) AS rn
,LEAD(value) OVER (PARTITION BY id ORDER BY sort_order) AS old_value
FROM CTE_YourQuery
)
SELECT
id
,new_value
,old_value
FROM CTE_RowNumber
WHERE rn = 1
ORDER BY id;
Assuming there is some other column which defines the "order" in which the new and old value appears, you can do this:
select t1.id, t1.value as old_value, t2.value as new_value
from the_table t1
join the_table t2 on t1.id = t2.id and t1.sort_order < t2.sort_order
But you have to have some column that distinguishes the row that is considered "old" from the one that is considered "new".