Related
generate_series function on Redshift works as expected, when used in a simple select statement.
WITH series AS (
SELECT n as id from generate_series (-10, 0, 1) n
) SELECT * FROM series;
-- Works fine
As soon as I add a JOIN condition, redshift throws
com.amazon.support.exceptions.ErrorException: Function
generate_series(integer,integer,integer)" not supported"
DROP TABLE testing;
CREATE TABLE testing (
id INT
);
WITH series AS (
SELECT n as id from generate_series (-10, 0, 1) n
) SELECT * FROM series S JOIN testing T ON S.id = T.id;
-- Function "generate_series(integer,integer,integer)" not supported.
Redshift Version
SELECT version();
-- PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.1485
Are there any workarounds to make this work?
generate_series is not supported by Redshift. It works only standalone on a leader node.
A workaround would be using row_number against any table that has sufficient number of rows:
with
series as (
select (row_number() over ())-11 from some_table limit 10
) ...
also, this question was asked multiple times already
You are correct that this does not work on Redshift.
See here.
The easiest workaround is to create a permanent table "manually" beforehand with the values within that table, e.g. you could have rows on that table for -1000 to +1000, then select the range from that table,
So for your example you would have something like
WITH series AS (
SELECT n as id from (select num as n from newtable where num between -10 and 0) n
) SELECT * FROM series S JOIN testing T ON S.id = T.id;
Does that work for you?
Alternatively, if you cannot create the table beforehand or prefer not to, you could use something like this
with ten_numbers as (select 1 as num union select 2 union select 3 union select 4 union select 5 union select 6 union select 7 union select 8 union select 9 union select 0)
,generted_numbers AS
(
SELECT (1000*t1.num) + (100*t2.num) + (10*t3.num) + t4.num-5000 as gen_num
FROM ten_numbers AS t1
JOIN ten_numbers AS t2 ON 1 = 1
JOIN ten_numbers AS t3 ON 1 = 1
JOIN ten_numbers AS t4 ON 1 = 1
)
select gen_num from generted_numbers
where gen_num between -10 and 0
order by 1;
I have table with a unique auto-incremental primary key. Over time, entries may be deleted from the table, so there are "holes" in this field's values. For example, table data may be as follows:
ID | Value | More fields...
---------------------------------
2 | Cat | ...
3 | Fish | ...
6 | Dog | ...
7 | Aardvark | ...
9 | Owl | ...
10 | Pig | ...
11 | Badger | ...
15 | Mongoose | ...
19 | Ferret | ...
I'm interested in a query that will return the list of missing IDs in the table. For the data above, the expected results are:
ID
----
1
4
5
8
12
13
14
16
17
18
Notes:
It is assumed that the initial first ID was 1
The maximum ID that should be examined is the final one, i.e. it's okay to assume that there were no additional entries after the current last one (see additional data on this point below)
A drawback of the above requirements is that the list will not return IDs that were created after ID 19 and that were deleted. I'm currently solving this case in code, because I hold the max ID created. However, if the query can take as a parameter MaxID, and also return those IDs between the current max and MaxID, that would be a nice "bonus" (but certainly not a must).
I'm currently working with MySQL, but consider moving to SQL Server, so I would like the query to fit both. Also, if you are using anything that can't run on SQLite, please mention it, thanks.
I landed on this page hoping to find a solution for SQLITE as this was the only answer I found when searching for this same question for SQLITE.
The final solution I found was from this article here
Float Middle Blog - SQLITE answer
Hope it helps someone else out :-)
the simple solution being:
SELECT DISTINCT id +1
FROM mytable
WHERE id + 1 NOT IN (SELECT DISTINCT id FROM mytable);
genius.
This question often comes up, and sadly, the most common (and most portable) answer is to create a temporary table to hold the IDs that should be there, and do a left join. The syntax is pretty similar between MySQL and SQL Server. The only real difference is the temporary tables syntax.
In MySQL:
declare #id int
declare #maxid int
set #id = 1
select #maxid = max(id) from tbl
create temporary table IDSeq
(
id int
)
while #id < #maxid
begin
insert into IDSeq values(#id)
set #id = #id + 1
end
select
s.id
from
idseq s
left join tbl t on
s.id = t.id
where t.id is null
drop table IDSeq
In SQL Server:
declare #id int
declare #maxid int
set #id = 1
select #maxid = max(id) from tbl
create table #IDSeq
(
id int
)
while #id < #maxid --whatever you max is
begin
insert into #IDSeq values(#id)
set #id = #id + 1
end
select
s.id
from
#idseq s
left join tbl t on
s.id = t.id
where t.id is null
drop table #IDSeq
Here's the query for SQL Server:
;WITH Missing (missnum, maxid)
AS
(
SELECT 1 AS missnum, (select max(id) from #TT)
UNION ALL
SELECT missnum + 1, maxid FROM Missing
WHERE missnum < maxid
)
SELECT missnum
FROM Missing
LEFT OUTER JOIN #TT tt on tt.id = Missing.missnum
WHERE tt.id is NULL
OPTION (MAXRECURSION 0);
Hope this is helpful.
PostgreSQL-only, inspired by other answers here.
SELECT all_ids AS missing_ids
FROM generate_series((SELECT MIN(id) FROM your_table), (SELECT MAX(id) FROM your_table)) all_ids
EXCEPT
SELECT id FROM your_table
I know it's an old question and already has an accepted answer,
but using a temp table isn't really necessary. Fixed formatting (sorry for double post).
DECLARE #TEST_ID integer, #LAST_ID integer, #ID integer
SET #TEST_ID = 1 -- start compare with this ID
SET #LAST_ID = 100 -- end compare with this ID
WHILE #TEST_ID <= #LAST_ID
BEGIN
SELECT #ID = (SELECT <column> FROM <table> WHERE <column> = #TEST_ID)
IF #ID IS NULL
BEGIN
PRINT 'Missing ID: ' + CAST(#TEST_ID AS VARCHAR(10))
END
SET #TEST_ID = #TEST_ID + 1
END
This is an Oracle only solution. It doesn't address the full question, but is left here for others that may be using Oracle.
select level id -- generate 1 .. 19
from dual
connect by level <= 19
minus -- remove from that set
select id -- everything that is currently in the
from table -- actual table
to get the missing rows from table
DECLARE #MaxID INT = (SELECT MAX(ID) FROM TABLE1)
SELECT SeqID AS MissingSeqID
FROM (SELECT ROW_NUMBER() OVER (ORDER BY column_id) SeqID from sys.columns) LkUp
LEFT JOIN dbo.TABLE1 t ON t.ID = LkUp.SeqID
WHERE t.ID is null and SeqID < #MaxID
I just have found the solution for Postgres:
select min(gs)
from generate_series(1, 1999) as gs
where gs not in (select id from mytable)
The single query can find the missing IDs..
SELECT distinct number
FROM master..spt_values
WHERE number BETWEEN 1 and (SELECT max(id) FROM MyTable)
AND number NOT IN (SELECT id FROM MyTable)
Update: This method took way too long so I wrote a linux command to find gaps in a text file. It does so in reverse order so first dump all id's to a text file like so;
nohup mysql --password=xx -e 'select id from tablename order by id desc' databasename > /home/ids.txt &
The first and last two lines are just to keep track of how long it took. 1.5million IDs(ish) took me 57sec & that's on a slow server. Set the max id in i and have at it.
T="$(date +%s)"; \
i=1574115; \
while read line; do \
if [[ "$line" != "$i" ]] ; then \
if [[ $i -lt 1 ]] ; then break; fi; \
if [[ $line -gt 1 ]] ; then \
missingsequenceend=$(( $line + 1 )); \
minusstr="-"; \
missingsequence="$missingsequenceend$minusstr$i"; \
expectnext=$(( $line - 1 )); \
i=$expectnext; \
echo -e "$missingsequence"; \
fi; \
else \
i=$(( $i - 1 )); \
fi; \
done \
< /home/ids.txt; \
T="$(($(date +%s)-T))"; \
echo "Time in seconds: ${T}"
Example output:
1494505-1494507
47566-47572
Time in seconds: 57
Also, I got syntax errors with the code from Eric's answer, but after changing the delimiter, using semicolons in the proper places and storing it in a procedure, it works.
Make sure you set the proper max ID, database name and table name (it's in the select query). And if you want to change the procedure name, change it in all 3 places.
use dbname;
drop procedure if exists dorepeat;
delimiter #
CREATE PROCEDURE dorepeat()
BEGIN
set #id = 1;
set #maxid = 1573736;
drop table if exists IDSeq;
create temporary table IDSeq
(
id int
);
WHILE #id < #maxid DO
insert into IDSeq values(#id);
set #id = #id + 1;
END WHILE;
select
s.id
from
IDSeq s
left join tablename t on
s.id = t.id
where t.id is null;
drop table if exists IDSeq;
END#
delimiter ;
CALL dorepeat;
I also found this query elwhere, but I haven't tested it.
SELECT a.id+1 AS start, MIN(b.id) - 1 AS end
FROM tablename AS a, tablename AS b
WHERE a.id < b.id
GROUP BY a.id
HAVING start < MIN(b.id)
TRY in MySQL
DELIMITER ||
DROP PROCEDURE IF EXISTS proc_missing ||
CREATE PROCEDURE proc_missing()
BEGIN
SET #minID = (SELECT MIN(`id`) FROM `tbl_name` WHERE `user_id`=13);
SET #maxID = (SELECT MAX(`id`) FROM `tbl_name` WHERE `user_id`=13);
REPEAT
SET #tableID = (SELECT `id` FROM `tbl_name` WHERE `id` = #minID);
IF (#tableID IS NULL) THEN
INSERT INTO temp_missing SET `missing_id` = #tableID;
END IF;
SET #minID = #minID + 1;
UNTIL(#minID <= #maxID)
END REPEAT;
END ||
DELIMITER ;
A few days ago, I was working on a production report and found some numbers missing. The missing numbers are very important, so I was asked to find a list of all missing numbers for investigation purposes. I posted a blog entry here, with a full demo, including a script to find missing numbers/IDs in a sample table.
The script suggested is quite long, so I won't include it here. Here are the basic steps used:
Create one temp table and store all distinct Numbers.
Find NextID which has something missing before it. Store into one TempTable.
Create one temp table to store missing number details.
Start to find the missing id using WHILE Loop.
Select missing data from #MissingID temp table.
Converting the SQL CTE (from Paul Svirin) to the Oracle version it looks like this (replace :YOURTABLE with the name of your table):
WITH Missing (missnum,maxid) as (
SELECT 1 missnum, (select max(id) from :YOURTABLE) maxid from dual
UNION ALL
SELECT m.missnum + 1,m.maxid
FROM Missing m
WHERE m.missnum < m.maxid
)
SELECT missnum
FROM Missing
LEFT OUTER JOIN :YOURTABLE tt on tt.id = Missing.missnum
WHERE tt.id is NULL
Using #PaulSvirin's answer, I've expanded it with a UNION to show ALL the data in my table, including the missing records with NULLs.
WITH Missing(missnum, maxid) AS
(SELECT (SELECT MIN(tmMIN.TETmeetingID)
FROM tblTETMeeting AS tmMIN)
AS missnum,
(SELECT MAX(tmMAX.TETmeetingID)
FROM tblTETMeeting AS tmMAX)
AS maxid
UNION ALL
SELECT missnum + 1, maxid
FROM Missing
WHERE missnum < maxid)
SELECT missnum AS TETmeetingID,
tt.DateID,
tt.WeekNo,
tt.TETID
FROM Missing LEFT JOIN tblTETMeeting tt ON tt.TETmeetingID = Missing.missnum
WHERE tt.TETmeetingID IS NULL
UNION
SELECT tt.TETmeetingID,
tt.DateID,
tt.WeekNo,
tt.TETID
FROM tblTETMeeting AS tt
OPTION ( MAXRECURSION 0 )
Work's great!
TETmeetingID DateID WeekNo TETID
29 3063 21 1
30 null null null
31 null null null
32 null null null
33 null null null
34 3070 22 1
35 3073 23 1
Easiest solution for me: Create a select that gives all ids up to max sequence value (ex:1000000), and filter:
with listids as (
Select Rownum idnumber From dual Connect By Rownum <= 1000000)
select * from listids
where idnumber not in (select id from table where id <=1000000)
A modified version borrowing #Eric proposal. This is for SQL Server and holds in a temp table the start and end value for missing ranges. If the gap is just one value it puts NULL as end value for easier visualization.
It will produce an output like this
|StartId| EndId |
|-------|-------|
| 1 | 10182 |
| 10189 | NULL |
| 10246 | 15000 |
And this is the script where myTable and id needs to be replaced by your table and identity column.
declare #id bigint
declare #endId bigint
declare #maxid bigint
declare #previousid bigint=0
set #id = 1
select #maxid = max(id) from myTable
create table #IDGaps
(
startId bigint,
endId bigint
)
while #id < #maxid
begin
if NOT EXISTS(select id from myTable where id=#id)
BEGIN
SET #previousid=#id
select top 1 #endId=id from myTable where id>#id
IF #id=#endId-1
insert into #IDGaps values(#id,null)
ELSE
insert into #IDGaps values(#id,#endId-1)
SET #id=#endId
END
ELSE
set #id = #id + 1
end
select * from #IDGaps
drop table #IDGaps
SOLUTION FOR SQLITE
if your table id only support positive values you can use this
SELECT DISTINCT table_id - 1 AS next_id
FROM table
WHERE next_id NOT IN (SELECT DISTINCT table_id FROM table)
AND next_id > 0
otherwise you should remove ids greater than the biggest id with
SELECT DISTINCT table_id + 1 AS next_id
FROM table
WHERE next_id NOT IN (SELECT DISTINCT table_id FROM table)
AND id < (SELECT MAX(id) FROM table)
This problem can be solved with only one query
select lft.id + 1 as missing_ids
from tbl as lft left outer join tbl as rght on lft.id + 1 = rght.id
where rght.id is null and lft.id between 1 and (Select max(id)-1 from tbl)
Tested on Mysql
Try This Query. This single query is enough to get missing numbers:(Please replace TABLE_NAME to which table name you are using)
select sno as missing from(SELECT #row := #row + 1 as sno FROM
(select 0 union all select 1 union all select 3 union all select 4 union all
select 5 union all select 6 union all select 6 union all select 7 union all
select 8 union all select 9) t,(select 0 union all select 1 union all select 3
union all select 4 union all select 5 union all select 6 union all select 6
union all select 7 union all select 8 union all select 9) t2,(select 0
union all select 1 union all select 3 union all select 4 union all select 5
union all select 6 union all select 6 union all select 7 union all select 8
union all select 9) t3, (select 0 union all select 1 union all select 3 union
all select 4 union all select 5 union all select 6 union all select 6 union all
select 7 union all select 8 union all select 9) t4,
(SELECT #row:=0) as b where #row<1000) as a where a.sno not in
(select distinct b.no from
(select b.*,if(#mn=0,#mn:=b.no,#mn) as min,(#mx:=b.no) as max from
(select ID as no from TABLE_NAME as a) as b,
(select #mn:=0,#mx:=0) as x order by no) as b) and
a.sno between #mn and #mx;
SELECT DISTINCT id -1
FROM users
WHERE id != 1 AND id - 1 NOT IN (SELECT DISTINCT id FROM users)
Explanation: ( id - 1 )..... checking for any previous id present in table
( id != 1 ).....neglecting when current id is 1 as its previous id will be 0 zero.
I have a large audit table and needed something that ran quickly - this worked well for me. It merges the top and bottom IDs for the missing ranges
select minQ.num,minId,maxId from
(SELECT DISTINCT id +1 as minId, Row_Number() Over ( Order By id ) As Num
FROM tblAuditLoghistory
WHERE id + 1 NOT IN (SELECT DISTINCT id FROM tblAuditLogHistory)
AND id < (SELECT max(id) FROM tblAuditLoghistory)) Minq
join
(SELECT DISTINCT id - 1 as maxId, Row_Number() Over ( Order By id ) As Num
FROM tblAuditLoghistory
WHERE id - 1 NOT IN (SELECT DISTINCT id FROM tblAuditLogHistory)
AND id > (SELECT min(id) FROM tblAuditLoghistory)) maxQ on minQ.num=maxQ.num
This what i used to find the missing id of one table named as tablename
select a.id+1 missing_ID from tablename a
where a.id+1 not in (select id from tablename b where b.id=a.id+1)
and a.id!=(select id from tablename c order by id desc limit 1)
It will return the missing ids.
If there are two(2) or more continuous missing ids, it will return only the first.
T-SQL query for finding first missing sequence string (prefix+no)
Sequence can have a prefix + a continuing no.
ex sequence will be
ID
-------
AUTO_500
AUTO_501
AUTO_502
AUTO_504
AUTO_505
AUTO_506
AUTO_507
AUTO_508
So above the missing sequence is AUTO_503 or if there is no missing sequence then it must return next sequence.
Also starting no is to specified ex. 500 in this case and prefix can be null i.e. no prefix only numbers as sequence.
You could LEFT JOIN the id numbers on shifted(+1) values to find gaps in sequential order:
SELECT
MIN(a.offsetnum) AS first_missing_num
FROM
(
SELECT 500 AS offsetnum
UNION
SELECT CAST(REPLACE(id, 'AUTO_', '') AS INT) + 1
FROM tbl
) a
LEFT JOIN
(SELECT CAST(REPLACE(id, 'AUTO_', '') AS INT) AS idnum FROM tbl) b ON a.offsetnum = b.idnum
WHERE
a.offsetnum >= 500 AND b.idnum IS NULL
SQLFiddle Demo
Using a recursive CTE to dynamically generate the sequence between the min and max of the ID Numbers maybe over complicated things a bit but it seems to work -
LIVE ON FIDDLE
CREATE TABLE tbl (
id VARCHAR(55)
);
INSERT INTO tbl VALUES
('AUTO_500'),
('AUTO_501'),
('AUTO_502'),
('AUTO_504'),
('AUTO_505'),
('AUTO_506'),
('AUTO_507'),
('AUTO_508'),
('509');
;WITH
data_cte(id)AS
(SELECT [id] = CAST(REPLACE(id, 'AUTO_', '') AS INT) FROM tbl)
,maxmin_cte(minId, maxId)AS
(SELECT [minId] = min(id),[maxId] = max(id) FROM data_cte)
,recursive_cte(n) AS
(
SELECT [minId] n from maxmin_cte
UNION ALL
SELECT (1 + n) n FROM recursive_cte WHERE n < (SELECT [maxId] from maxmin_cte)
)
SELECT x.n
FROM
recursive_cte x
LEFT OUTER JOIN data_cte y ON
x.n = y.id
WHERE y.id IS NULL
Check this solution.Here you just need to add identity column.
CREATE TABLE tbl (
id VARCHAR(55),
idn int identity(0,1)
);
INSERT INTO tbl VALUES
('AUTO_500'),
('AUTO_501'),
('AUTO_502'),
('AUTO_504'),
('AUTO_505'),
('AUTO_506'),
('AUTO_507'),
('AUTO_508'),
('509');
SELECT min(idn+500) FROM tbl where 'AUTO_'+cast((idn+500) as varchar)<>id
try this:
with cte as(
select cast(REPLACE(id,'AUTO_','') as int)-500+1 [diff],ROW_NUMBER()
over(order by cast(REPLACE(id,'AUTO_','') as int)) [rnk] from tbl)
select top 1 'AUTO_'+cast(500+rnk as varchar(50)) [ID] from cte
where [diff]=[rnk]
order by rnk desc
SQL FIddle Demo
Had a similar situation, where we have R_Cds that were like this R01005
;with Active_R_CD (R_CD)
As
(
Select Distinct Cast(Replace(R_CD,'R', ' ') as Int)
from table
where stat = 1)
select Arc.R_CD + 1 as 'Gaps in R Code'
from Active_R_CD as Arc
left outer join Active_R_CD as r on ARC.R_CD + 1 = R.R_CD
where R.R_CD is null
order by 1
I'd like to find the first "gap" in a counter column in an SQL table. For example, if there are values 1,2,4 and 5 I'd like to find out 3.
I can of course get the values in order and go through it manually, but I'd like to know if there would be a way to do it in SQL.
In addition, it should be quite standard SQL, working with different DBMSes.
In MySQL and PostgreSQL:
SELECT id + 1
FROM mytable mo
WHERE NOT EXISTS
(
SELECT NULL
FROM mytable mi
WHERE mi.id = mo.id + 1
)
ORDER BY
id
LIMIT 1
In SQL Server:
SELECT TOP 1
id + 1
FROM mytable mo
WHERE NOT EXISTS
(
SELECT NULL
FROM mytable mi
WHERE mi.id = mo.id + 1
)
ORDER BY
id
In Oracle:
SELECT *
FROM (
SELECT id + 1 AS gap
FROM mytable mo
WHERE NOT EXISTS
(
SELECT NULL
FROM mytable mi
WHERE mi.id = mo.id + 1
)
ORDER BY
id
)
WHERE rownum = 1
ANSI (works everywhere, least efficient):
SELECT MIN(id) + 1
FROM mytable mo
WHERE NOT EXISTS
(
SELECT NULL
FROM mytable mi
WHERE mi.id = mo.id + 1
)
Systems supporting sliding window functions:
SELECT -- TOP 1
-- Uncomment above for SQL Server 2012+
previd
FROM (
SELECT id,
LAG(id) OVER (ORDER BY id) previd
FROM mytable
) q
WHERE previd <> id - 1
ORDER BY
id
-- LIMIT 1
-- Uncomment above for PostgreSQL
Your answers all work fine if you have a first value id = 1, otherwise this gap will not be detected. For instance if your table id values are 3,4,5, your queries will return 6.
I did something like this
SELECT MIN(ID+1) FROM (
SELECT 0 AS ID UNION ALL
SELECT
MIN(ID + 1)
FROM
TableX) AS T1
WHERE
ID+1 NOT IN (SELECT ID FROM TableX)
There isn't really an extremely standard SQL way to do this, but with some form of limiting clause you can do
SELECT `table`.`num` + 1
FROM `table`
LEFT JOIN `table` AS `alt`
ON `alt`.`num` = `table`.`num` + 1
WHERE `alt`.`num` IS NULL
LIMIT 1
(MySQL, PostgreSQL)
or
SELECT TOP 1 `num` + 1
FROM `table`
LEFT JOIN `table` AS `alt`
ON `alt`.`num` = `table`.`num` + 1
WHERE `alt`.`num` IS NULL
(SQL Server)
or
SELECT `num` + 1
FROM `table`
LEFT JOIN `table` AS `alt`
ON `alt`.`num` = `table`.`num` + 1
WHERE `alt`.`num` IS NULL
AND ROWNUM = 1
(Oracle)
The first thing that came into my head. Not sure if it's a good idea to go this way at all, but should work. Suppose the table is t and the column is c:
SELECT
t1.c + 1 AS gap
FROM t as t1
LEFT OUTER JOIN t as t2 ON (t1.c + 1 = t2.c)
WHERE t2.c IS NULL
ORDER BY gap ASC
LIMIT 1
Edit: This one may be a tick faster (and shorter!):
SELECT
min(t1.c) + 1 AS gap
FROM t as t1
LEFT OUTER JOIN t as t2 ON (t1.c + 1 = t2.c)
WHERE t2.c IS NULL
This works in SQL Server - can't test it in other systems but it seems standard...
SELECT MIN(t1.ID)+1 FROM mytable t1 WHERE NOT EXISTS (SELECT ID FROM mytable WHERE ID = (t1.ID + 1))
You could also add a starting point to the where clause...
SELECT MIN(t1.ID)+1 FROM mytable t1 WHERE NOT EXISTS (SELECT ID FROM mytable WHERE ID = (t1.ID + 1)) AND ID > 2000
So if you had 2000, 2001, 2002, and 2005 where 2003 and 2004 didn't exist, it would return 2003.
The following solution:
provides test data;
an inner query that produces other gaps; and
it works in SQL Server 2012.
Numbers the ordered rows sequentially in the "with" clause and then reuses the result twice with an inner join on the row number, but offset by 1 so as to compare the row before with the row after, looking for IDs with a gap greater than 1. More than asked for but more widely applicable.
create table #ID ( id integer );
insert into #ID values (1),(2), (4),(5),(6),(7),(8), (12),(13),(14),(15);
with Source as (
select
row_number()over ( order by A.id ) as seq
,A.id as id
from #ID as A WITH(NOLOCK)
)
Select top 1 gap_start from (
Select
(J.id+1) as gap_start
,(K.id-1) as gap_end
from Source as J
inner join Source as K
on (J.seq+1) = K.seq
where (J.id - (K.id-1)) <> 0
) as G
The inner query produces:
gap_start gap_end
3 3
9 11
The outer query produces:
gap_start
3
Inner join to a view or sequence that has a all possible values.
No table? Make a table. I always keep a dummy table around just for this.
create table artificial_range(
id int not null primary key auto_increment,
name varchar( 20 ) null ) ;
-- or whatever your database requires for an auto increment column
insert into artificial_range( name ) values ( null )
-- create one row.
insert into artificial_range( name ) select name from artificial_range;
-- you now have two rows
insert into artificial_range( name ) select name from artificial_range;
-- you now have four rows
insert into artificial_range( name ) select name from artificial_range;
-- you now have eight rows
--etc.
insert into artificial_range( name ) select name from artificial_range;
-- you now have 1024 rows, with ids 1-1024
Then,
select a.id from artificial_range a
where not exists ( select * from your_table b
where b.counter = a.id) ;
This one accounts for everything mentioned so far. It includes 0 as a starting point, which it will default to if no values exist as well. I also added the appropriate locations for the other parts of a multi-value key. This has only been tested on SQL Server.
select
MIN(ID)
from (
select
0 ID
union all
select
[YourIdColumn]+1
from
[YourTable]
where
--Filter the rest of your key--
) foo
left join
[YourTable]
on [YourIdColumn]=ID
and --Filter the rest of your key--
where
[YourIdColumn] is null
For PostgreSQL
An example that makes use of recursive query.
This might be useful if you want to find a gap in a specific range
(it will work even if the table is empty, whereas the other examples will not)
WITH
RECURSIVE a(id) AS (VALUES (1) UNION ALL SELECT id + 1 FROM a WHERE id < 100), -- range 1..100
b AS (SELECT id FROM my_table) -- your table ID list
SELECT a.id -- find numbers from the range that do not exist in main table
FROM a
LEFT JOIN b ON b.id = a.id
WHERE b.id IS NULL
-- LIMIT 1 -- uncomment if only the first value is needed
My guess:
SELECT MIN(p1.field) + 1 as gap
FROM table1 AS p1
INNER JOIN table1 as p3 ON (p1.field = p3.field + 2)
LEFT OUTER JOIN table1 AS p2 ON (p1.field = p2.field + 1)
WHERE p2.field is null;
I wrote up a quick way of doing it. Not sure this is the most efficient, but gets the job done. Note that it does not tell you the gap, but tells you the id before and after the gap (keep in mind the gap could be multiple values, so for example 1,2,4,7,11 etc)
I'm using sqlite as an example
If this is your table structure
create table sequential(id int not null, name varchar(10) null);
and these are your rows
id|name
1|one
2|two
4|four
5|five
9|nine
The query is
select a.* from sequential a left join sequential b on a.id = b.id + 1 where b.id is null and a.id <> (select min(id) from sequential)
union
select a.* from sequential a left join sequential b on a.id = b.id - 1 where b.id is null and a.id <> (select max(id) from sequential);
https://gist.github.com/wkimeria/7787ffe84d1c54216f1b320996b17b7e
Here is an alternative to show the range of all possible gap values in portable and more compact way :
Assume your table schema looks like this :
> SELECT id FROM your_table;
+-----+
| id |
+-----+
| 90 |
| 103 |
| 104 |
| 118 |
| 119 |
| 120 |
| 121 |
| 161 |
| 162 |
| 163 |
| 185 |
+-----+
To fetch the ranges of all possible gap values, you have the following query :
The subquery lists pairs of ids, each of which has the lowerbound column being smaller than upperbound column, then use GROUP BY and MIN(m2.id) to reduce number of useless records.
The outer query further removes the records where lowerbound is exactly upperbound - 1
My query doesn't (explicitly) output the 2 records (YOUR_MIN_ID_VALUE, 89) and (186, YOUR_MAX_ID_VALUE) at both ends, that implicitly means any number in both of the ranges hasn't been used in your_table so far.
> SELECT m3.lowerbound + 1, m3.upperbound - 1 FROM
(
SELECT m1.id as lowerbound, MIN(m2.id) as upperbound FROM
your_table m1 INNER JOIN your_table
AS m2 ON m1.id < m2.id GROUP BY m1.id
)
m3 WHERE m3.lowerbound < m3.upperbound - 1;
+-------------------+-------------------+
| m3.lowerbound + 1 | m3.upperbound - 1 |
+-------------------+-------------------+
| 91 | 102 |
| 105 | 117 |
| 122 | 160 |
| 164 | 184 |
+-------------------+-------------------+
select min([ColumnName]) from [TableName]
where [ColumnName]-1 not in (select [ColumnName] from [TableName])
and [ColumnName] <> (select min([ColumnName]) from [TableName])
Here is standard a SQL solution that runs on all database servers with no change:
select min(counter + 1) FIRST_GAP
from my_table a
where not exists (select 'x' from my_table b where b.counter = a.counter + 1)
and a.counter <> (select max(c.counter) from my_table c);
See in action for;
PL/SQL via Oracle's livesql,
MySQL via sqlfiddle,
PostgreSQL via sqlfiddle
MS Sql via sqlfiddle
It works for empty tables or with negatives values as well. Just tested in SQL Server 2012
select min(n) from (
select case when lead(i,1,0) over(order by i)>i+1 then i+1 else null end n from MyTable) w
If You use Firebird 3 this is most elegant and simple:
select RowID
from (
select `ID_Column`, Row_Number() over(order by `ID_Column`) as RowID
from `Your_Table`
order by `ID_Column`)
where `ID_Column` <> RowID
rows 1
-- PUT THE TABLE NAME AND COLUMN NAME BELOW
-- IN MY EXAMPLE, THE TABLE NAME IS = SHOW_GAPS AND COLUMN NAME IS = ID
-- PUT THESE TWO VALUES AND EXECUTE THE QUERY
DECLARE #TABLE_NAME VARCHAR(100) = 'SHOW_GAPS'
DECLARE #COLUMN_NAME VARCHAR(100) = 'ID'
DECLARE #SQL VARCHAR(MAX)
SET #SQL =
'SELECT TOP 1
'+#COLUMN_NAME+' + 1
FROM '+#TABLE_NAME+' mo
WHERE NOT EXISTS
(
SELECT NULL
FROM '+#TABLE_NAME+' mi
WHERE mi.'+#COLUMN_NAME+' = mo.'+#COLUMN_NAME+' + 1
)
ORDER BY
'+#COLUMN_NAME
-- SELECT #SQL
DECLARE #MISSING_ID TABLE (ID INT)
INSERT INTO #MISSING_ID
EXEC (#SQL)
--select * from #MISSING_ID
declare #var_for_cursor int
DECLARE #LOW INT
DECLARE #HIGH INT
DECLARE #FINAL_RANGE TABLE (LOWER_MISSING_RANGE INT, HIGHER_MISSING_RANGE INT)
DECLARE IdentityGapCursor CURSOR FOR
select * from #MISSING_ID
ORDER BY 1;
open IdentityGapCursor
fetch next from IdentityGapCursor
into #var_for_cursor
WHILE ##FETCH_STATUS = 0
BEGIN
SET #SQL = '
DECLARE #LOW INT
SELECT #LOW = MAX('+#COLUMN_NAME+') + 1 FROM '+#TABLE_NAME
+' WHERE '+#COLUMN_NAME+' < ' + cast( #var_for_cursor as VARCHAR(MAX))
SET #SQL = #sql + '
DECLARE #HIGH INT
SELECT #HIGH = MIN('+#COLUMN_NAME+') - 1 FROM '+#TABLE_NAME
+' WHERE '+#COLUMN_NAME+' > ' + cast( #var_for_cursor as VARCHAR(MAX))
SET #SQL = #sql + 'SELECT #LOW,#HIGH'
INSERT INTO #FINAL_RANGE
EXEC( #SQL)
fetch next from IdentityGapCursor
into #var_for_cursor
END
CLOSE IdentityGapCursor;
DEALLOCATE IdentityGapCursor;
SELECT ROW_NUMBER() OVER(ORDER BY LOWER_MISSING_RANGE) AS 'Gap Number',* FROM #FINAL_RANGE
Found most of approaches run very, very slow in mysql. Here is my solution for mysql < 8.0. Tested on 1M records with a gap near the end ~ 1sec to finish. Not sure if it fits other SQL flavours.
SELECT cardNumber - 1
FROM
(SELECT #row_number := 0) as t,
(
SELECT (#row_number:=#row_number+1), cardNumber, cardNumber-#row_number AS diff
FROM cards
ORDER BY cardNumber
) as x
WHERE diff >= 1
LIMIT 0,1
I assume that sequence starts from `1`.
If your counter is starting from 1 and you want to generate first number of sequence (1) when empty, here is the corrected piece of code from first answer valid for Oracle:
SELECT
NVL(MIN(id + 1),1) AS gap
FROM
mytable mo
WHERE 1=1
AND NOT EXISTS
(
SELECT NULL
FROM mytable mi
WHERE mi.id = mo.id + 1
)
AND EXISTS
(
SELECT NULL
FROM mytable mi
WHERE mi.id = 1
)
DECLARE #Table AS TABLE(
[Value] int
)
INSERT INTO #Table ([Value])
VALUES
(1),(2),(4),(5),(6),(10),(20),(21),(22),(50),(51),(52),(53),(54),(55)
--Gaps
--Start End Size
--3 3 1
--7 9 3
--11 19 9
--23 49 27
SELECT [startTable].[Value]+1 [Start]
,[EndTable].[Value]-1 [End]
,([EndTable].[Value]-1) - ([startTable].[Value]) Size
FROM
(
SELECT [Value]
,ROW_NUMBER() OVER(PARTITION BY 1 ORDER BY [Value]) Record
FROM #Table
)AS startTable
JOIN
(
SELECT [Value]
,ROW_NUMBER() OVER(PARTITION BY 1 ORDER BY [Value]) Record
FROM #Table
)AS EndTable
ON [EndTable].Record = [startTable].Record+1
WHERE [startTable].[Value]+1 <>[EndTable].[Value]
If the numbers in the column are positive integers (starting from 1) then here is how to solve it easily. (assuming ID is your column name)
SELECT TEMP.ID
FROM (SELECT ROW_NUMBER() OVER () AS NUM FROM 'TABLE-NAME') AS TEMP
WHERE ID NOT IN (SELECT ID FROM 'TABLE-NAME')
ORDER BY 1 ASC LIMIT 1
SELECT ID+1 FROM table WHERE ID+1 NOT IN (SELECT ID FROM table) ORDER BY 1;
I have table with a unique auto-incremental primary key. Over time, entries may be deleted from the table, so there are "holes" in this field's values. For example, table data may be as follows:
ID | Value | More fields...
---------------------------------
2 | Cat | ...
3 | Fish | ...
6 | Dog | ...
7 | Aardvark | ...
9 | Owl | ...
10 | Pig | ...
11 | Badger | ...
15 | Mongoose | ...
19 | Ferret | ...
I'm interested in a query that will return the list of missing IDs in the table. For the data above, the expected results are:
ID
----
1
4
5
8
12
13
14
16
17
18
Notes:
It is assumed that the initial first ID was 1
The maximum ID that should be examined is the final one, i.e. it's okay to assume that there were no additional entries after the current last one (see additional data on this point below)
A drawback of the above requirements is that the list will not return IDs that were created after ID 19 and that were deleted. I'm currently solving this case in code, because I hold the max ID created. However, if the query can take as a parameter MaxID, and also return those IDs between the current max and MaxID, that would be a nice "bonus" (but certainly not a must).
I'm currently working with MySQL, but consider moving to SQL Server, so I would like the query to fit both. Also, if you are using anything that can't run on SQLite, please mention it, thanks.
I landed on this page hoping to find a solution for SQLITE as this was the only answer I found when searching for this same question for SQLITE.
The final solution I found was from this article here
Float Middle Blog - SQLITE answer
Hope it helps someone else out :-)
the simple solution being:
SELECT DISTINCT id +1
FROM mytable
WHERE id + 1 NOT IN (SELECT DISTINCT id FROM mytable);
genius.
This question often comes up, and sadly, the most common (and most portable) answer is to create a temporary table to hold the IDs that should be there, and do a left join. The syntax is pretty similar between MySQL and SQL Server. The only real difference is the temporary tables syntax.
In MySQL:
declare #id int
declare #maxid int
set #id = 1
select #maxid = max(id) from tbl
create temporary table IDSeq
(
id int
)
while #id < #maxid
begin
insert into IDSeq values(#id)
set #id = #id + 1
end
select
s.id
from
idseq s
left join tbl t on
s.id = t.id
where t.id is null
drop table IDSeq
In SQL Server:
declare #id int
declare #maxid int
set #id = 1
select #maxid = max(id) from tbl
create table #IDSeq
(
id int
)
while #id < #maxid --whatever you max is
begin
insert into #IDSeq values(#id)
set #id = #id + 1
end
select
s.id
from
#idseq s
left join tbl t on
s.id = t.id
where t.id is null
drop table #IDSeq
Here's the query for SQL Server:
;WITH Missing (missnum, maxid)
AS
(
SELECT 1 AS missnum, (select max(id) from #TT)
UNION ALL
SELECT missnum + 1, maxid FROM Missing
WHERE missnum < maxid
)
SELECT missnum
FROM Missing
LEFT OUTER JOIN #TT tt on tt.id = Missing.missnum
WHERE tt.id is NULL
OPTION (MAXRECURSION 0);
Hope this is helpful.
PostgreSQL-only, inspired by other answers here.
SELECT all_ids AS missing_ids
FROM generate_series((SELECT MIN(id) FROM your_table), (SELECT MAX(id) FROM your_table)) all_ids
EXCEPT
SELECT id FROM your_table
I know it's an old question and already has an accepted answer,
but using a temp table isn't really necessary. Fixed formatting (sorry for double post).
DECLARE #TEST_ID integer, #LAST_ID integer, #ID integer
SET #TEST_ID = 1 -- start compare with this ID
SET #LAST_ID = 100 -- end compare with this ID
WHILE #TEST_ID <= #LAST_ID
BEGIN
SELECT #ID = (SELECT <column> FROM <table> WHERE <column> = #TEST_ID)
IF #ID IS NULL
BEGIN
PRINT 'Missing ID: ' + CAST(#TEST_ID AS VARCHAR(10))
END
SET #TEST_ID = #TEST_ID + 1
END
This is an Oracle only solution. It doesn't address the full question, but is left here for others that may be using Oracle.
select level id -- generate 1 .. 19
from dual
connect by level <= 19
minus -- remove from that set
select id -- everything that is currently in the
from table -- actual table
to get the missing rows from table
DECLARE #MaxID INT = (SELECT MAX(ID) FROM TABLE1)
SELECT SeqID AS MissingSeqID
FROM (SELECT ROW_NUMBER() OVER (ORDER BY column_id) SeqID from sys.columns) LkUp
LEFT JOIN dbo.TABLE1 t ON t.ID = LkUp.SeqID
WHERE t.ID is null and SeqID < #MaxID
I just have found the solution for Postgres:
select min(gs)
from generate_series(1, 1999) as gs
where gs not in (select id from mytable)
The single query can find the missing IDs..
SELECT distinct number
FROM master..spt_values
WHERE number BETWEEN 1 and (SELECT max(id) FROM MyTable)
AND number NOT IN (SELECT id FROM MyTable)
Update: This method took way too long so I wrote a linux command to find gaps in a text file. It does so in reverse order so first dump all id's to a text file like so;
nohup mysql --password=xx -e 'select id from tablename order by id desc' databasename > /home/ids.txt &
The first and last two lines are just to keep track of how long it took. 1.5million IDs(ish) took me 57sec & that's on a slow server. Set the max id in i and have at it.
T="$(date +%s)"; \
i=1574115; \
while read line; do \
if [[ "$line" != "$i" ]] ; then \
if [[ $i -lt 1 ]] ; then break; fi; \
if [[ $line -gt 1 ]] ; then \
missingsequenceend=$(( $line + 1 )); \
minusstr="-"; \
missingsequence="$missingsequenceend$minusstr$i"; \
expectnext=$(( $line - 1 )); \
i=$expectnext; \
echo -e "$missingsequence"; \
fi; \
else \
i=$(( $i - 1 )); \
fi; \
done \
< /home/ids.txt; \
T="$(($(date +%s)-T))"; \
echo "Time in seconds: ${T}"
Example output:
1494505-1494507
47566-47572
Time in seconds: 57
Also, I got syntax errors with the code from Eric's answer, but after changing the delimiter, using semicolons in the proper places and storing it in a procedure, it works.
Make sure you set the proper max ID, database name and table name (it's in the select query). And if you want to change the procedure name, change it in all 3 places.
use dbname;
drop procedure if exists dorepeat;
delimiter #
CREATE PROCEDURE dorepeat()
BEGIN
set #id = 1;
set #maxid = 1573736;
drop table if exists IDSeq;
create temporary table IDSeq
(
id int
);
WHILE #id < #maxid DO
insert into IDSeq values(#id);
set #id = #id + 1;
END WHILE;
select
s.id
from
IDSeq s
left join tablename t on
s.id = t.id
where t.id is null;
drop table if exists IDSeq;
END#
delimiter ;
CALL dorepeat;
I also found this query elwhere, but I haven't tested it.
SELECT a.id+1 AS start, MIN(b.id) - 1 AS end
FROM tablename AS a, tablename AS b
WHERE a.id < b.id
GROUP BY a.id
HAVING start < MIN(b.id)
TRY in MySQL
DELIMITER ||
DROP PROCEDURE IF EXISTS proc_missing ||
CREATE PROCEDURE proc_missing()
BEGIN
SET #minID = (SELECT MIN(`id`) FROM `tbl_name` WHERE `user_id`=13);
SET #maxID = (SELECT MAX(`id`) FROM `tbl_name` WHERE `user_id`=13);
REPEAT
SET #tableID = (SELECT `id` FROM `tbl_name` WHERE `id` = #minID);
IF (#tableID IS NULL) THEN
INSERT INTO temp_missing SET `missing_id` = #tableID;
END IF;
SET #minID = #minID + 1;
UNTIL(#minID <= #maxID)
END REPEAT;
END ||
DELIMITER ;
A few days ago, I was working on a production report and found some numbers missing. The missing numbers are very important, so I was asked to find a list of all missing numbers for investigation purposes. I posted a blog entry here, with a full demo, including a script to find missing numbers/IDs in a sample table.
The script suggested is quite long, so I won't include it here. Here are the basic steps used:
Create one temp table and store all distinct Numbers.
Find NextID which has something missing before it. Store into one TempTable.
Create one temp table to store missing number details.
Start to find the missing id using WHILE Loop.
Select missing data from #MissingID temp table.
Converting the SQL CTE (from Paul Svirin) to the Oracle version it looks like this (replace :YOURTABLE with the name of your table):
WITH Missing (missnum,maxid) as (
SELECT 1 missnum, (select max(id) from :YOURTABLE) maxid from dual
UNION ALL
SELECT m.missnum + 1,m.maxid
FROM Missing m
WHERE m.missnum < m.maxid
)
SELECT missnum
FROM Missing
LEFT OUTER JOIN :YOURTABLE tt on tt.id = Missing.missnum
WHERE tt.id is NULL
Using #PaulSvirin's answer, I've expanded it with a UNION to show ALL the data in my table, including the missing records with NULLs.
WITH Missing(missnum, maxid) AS
(SELECT (SELECT MIN(tmMIN.TETmeetingID)
FROM tblTETMeeting AS tmMIN)
AS missnum,
(SELECT MAX(tmMAX.TETmeetingID)
FROM tblTETMeeting AS tmMAX)
AS maxid
UNION ALL
SELECT missnum + 1, maxid
FROM Missing
WHERE missnum < maxid)
SELECT missnum AS TETmeetingID,
tt.DateID,
tt.WeekNo,
tt.TETID
FROM Missing LEFT JOIN tblTETMeeting tt ON tt.TETmeetingID = Missing.missnum
WHERE tt.TETmeetingID IS NULL
UNION
SELECT tt.TETmeetingID,
tt.DateID,
tt.WeekNo,
tt.TETID
FROM tblTETMeeting AS tt
OPTION ( MAXRECURSION 0 )
Work's great!
TETmeetingID DateID WeekNo TETID
29 3063 21 1
30 null null null
31 null null null
32 null null null
33 null null null
34 3070 22 1
35 3073 23 1
Easiest solution for me: Create a select that gives all ids up to max sequence value (ex:1000000), and filter:
with listids as (
Select Rownum idnumber From dual Connect By Rownum <= 1000000)
select * from listids
where idnumber not in (select id from table where id <=1000000)
A modified version borrowing #Eric proposal. This is for SQL Server and holds in a temp table the start and end value for missing ranges. If the gap is just one value it puts NULL as end value for easier visualization.
It will produce an output like this
|StartId| EndId |
|-------|-------|
| 1 | 10182 |
| 10189 | NULL |
| 10246 | 15000 |
And this is the script where myTable and id needs to be replaced by your table and identity column.
declare #id bigint
declare #endId bigint
declare #maxid bigint
declare #previousid bigint=0
set #id = 1
select #maxid = max(id) from myTable
create table #IDGaps
(
startId bigint,
endId bigint
)
while #id < #maxid
begin
if NOT EXISTS(select id from myTable where id=#id)
BEGIN
SET #previousid=#id
select top 1 #endId=id from myTable where id>#id
IF #id=#endId-1
insert into #IDGaps values(#id,null)
ELSE
insert into #IDGaps values(#id,#endId-1)
SET #id=#endId
END
ELSE
set #id = #id + 1
end
select * from #IDGaps
drop table #IDGaps
SOLUTION FOR SQLITE
if your table id only support positive values you can use this
SELECT DISTINCT table_id - 1 AS next_id
FROM table
WHERE next_id NOT IN (SELECT DISTINCT table_id FROM table)
AND next_id > 0
otherwise you should remove ids greater than the biggest id with
SELECT DISTINCT table_id + 1 AS next_id
FROM table
WHERE next_id NOT IN (SELECT DISTINCT table_id FROM table)
AND id < (SELECT MAX(id) FROM table)
This problem can be solved with only one query
select lft.id + 1 as missing_ids
from tbl as lft left outer join tbl as rght on lft.id + 1 = rght.id
where rght.id is null and lft.id between 1 and (Select max(id)-1 from tbl)
Tested on Mysql
Try This Query. This single query is enough to get missing numbers:(Please replace TABLE_NAME to which table name you are using)
select sno as missing from(SELECT #row := #row + 1 as sno FROM
(select 0 union all select 1 union all select 3 union all select 4 union all
select 5 union all select 6 union all select 6 union all select 7 union all
select 8 union all select 9) t,(select 0 union all select 1 union all select 3
union all select 4 union all select 5 union all select 6 union all select 6
union all select 7 union all select 8 union all select 9) t2,(select 0
union all select 1 union all select 3 union all select 4 union all select 5
union all select 6 union all select 6 union all select 7 union all select 8
union all select 9) t3, (select 0 union all select 1 union all select 3 union
all select 4 union all select 5 union all select 6 union all select 6 union all
select 7 union all select 8 union all select 9) t4,
(SELECT #row:=0) as b where #row<1000) as a where a.sno not in
(select distinct b.no from
(select b.*,if(#mn=0,#mn:=b.no,#mn) as min,(#mx:=b.no) as max from
(select ID as no from TABLE_NAME as a) as b,
(select #mn:=0,#mx:=0) as x order by no) as b) and
a.sno between #mn and #mx;
SELECT DISTINCT id -1
FROM users
WHERE id != 1 AND id - 1 NOT IN (SELECT DISTINCT id FROM users)
Explanation: ( id - 1 )..... checking for any previous id present in table
( id != 1 ).....neglecting when current id is 1 as its previous id will be 0 zero.
I have a large audit table and needed something that ran quickly - this worked well for me. It merges the top and bottom IDs for the missing ranges
select minQ.num,minId,maxId from
(SELECT DISTINCT id +1 as minId, Row_Number() Over ( Order By id ) As Num
FROM tblAuditLoghistory
WHERE id + 1 NOT IN (SELECT DISTINCT id FROM tblAuditLogHistory)
AND id < (SELECT max(id) FROM tblAuditLoghistory)) Minq
join
(SELECT DISTINCT id - 1 as maxId, Row_Number() Over ( Order By id ) As Num
FROM tblAuditLoghistory
WHERE id - 1 NOT IN (SELECT DISTINCT id FROM tblAuditLogHistory)
AND id > (SELECT min(id) FROM tblAuditLoghistory)) maxQ on minQ.num=maxQ.num
This what i used to find the missing id of one table named as tablename
select a.id+1 missing_ID from tablename a
where a.id+1 not in (select id from tablename b where b.id=a.id+1)
and a.id!=(select id from tablename c order by id desc limit 1)
It will return the missing ids.
If there are two(2) or more continuous missing ids, it will return only the first.