I have requirement where in I am creating a table dynamically(number of columns may change based on input parameter to my procedure through which I am creating this table) with data in the table like below.
PK col1 col2 col3
A null 1-2 3-4
B null null 4-5
C null 5-6 null
Now the requirement is I want to extract only the columns where at least there should be 1 record without null and spool the whole data into a file. My output should be like below (col1 exlluded from output as it has all nulls).
PK col2 col3
A 1-2 3-4
B null 4-5
C 5-6 null
Can anybody provide any hints to achieve this. Thanks in advance.
This won't be very efficient I suspect, but you can use COUNT() to determine if there are any only NULLS in a column because COUNT(column_here) will only add 1 for each non-null value. Hence if the count is zero that column is only NULLs.
This can then be combined into a query to generate a valid select statement and then that executed immediate (being careful of course to avoid sql injection).
Anyway, here's an example:
select
'select '
|| substr((
select
case when count(COL1) > 0 then ',col1' else '' end
|| case when count(COL2) > 0 then ',col2' else '' end
|| case when count(COL3) > 0 then ',col3' else '' end
from a_table
),2,8000)
|| ' from '
|| ' a_table'
as sql_string
from dual
;
see this sqlfiddle
result by the above is:
| SQL_STRING |
|--------------------------------|
| select col2,col3 from a_table |
Here is a try. First, create a function to generate your query, returning a REF CURSOR:
create or replace function select_non_nulls() return sys_refcursor as
myQuery varchar2(500);
myCur sys_refcursor;
begin
select 'select ' || listagg(col, ', ') within group (order by col) || ' from test'
into myQuery
from
(
select case when max(col1) is null then null else 'col1' end col from test
union all
select case when max(col2) is null then null else 'col2' end col from test
union all
select case when max(col3) is null then null else 'col3' end col from test
)
;
open myCur for myQuery;
return myCur;
end;
/
Then use it in SQL*Plus:
SQL> var rc refcursor
SQL> exec :rc := select_non_nulls;
SQL> print rc;
I used num_nulls from all_tab_cols and achieved result as per my requirement. Thank you.
Related
Hi I want to convert the sql statement into string for dynamic use. But I’m having trouble while concatenation
select
col1, col2
from
Table1
Where
Col3 in ( ‘cat’ , ‘dog’ );
I’m unable to figure out how to put quotes and || for the cat and dog
‘select
col1, col2
from
Table1
Where
Col3 in ( ‘||’‘cat’’ ||’, ‘’dog’ ‘||’)’ || Strcond;
One option is to use the q-quoting mechanism (see line #6), as it lets you use single quotes normally (also, your single quotes look fancy; if you planned to copy/paste from e.g. MS Word into your SQL client, that won't work).
Here's an example:
SQL> declare
2 l_str varchar2(500);
3 strcond varchar2(100) := ' and 1 = 1';
4 begin
5 l_str := 'select col1, col2 from table1 where col3 in ' ||
6 q'[('cat', 'dog')]' || strcond;
7 dbms_output.put_line(l_str);
8 end;
9 /
select col1, col2 from table1 where col3 in ('cat', 'dog') and 1 = 1
PL/SQL procedure successfully completed.
SQL>
One option it to use bind variables and put in as many bind variables as the maximum size of the IN list:
select col1, col2 from Table1 Where Col3 in ( :a001, :a002, :a003, :a004 )
Then:
you can use the same statement repeatedly and the SQL engine will not have to reparse it (after the first time).
you do not need to escape quotes in your list values in the dynamic SQL string.
your code is less vulnerable to SQL injection.
If you want to pass fewer values than the maximum then you can repeat values:
DECLARE
sql VARCHAR2(2000) := 'select col1, col2 from Table1 Where Col3 in ( :a001, :a002, :a003, :a004 )';
BEGIN
EXECUTE IMMEDIATE sql USING 'cat', 'dog', 'cat', 'cat';
END;
/
However, if you are going to do this then you can consider if you can eliminate dynamic SQL entirely:
DECLARE
a001 Table1.Col3%TYPE := 'cat';
a002 Table1.Col3%TYPE := 'dog';
a003 Table1.Col3%TYPE := a001;
a004 Table1.Col3%TYPE := a001;
col1_values SYS.ODCIVARCHAR2LIST;
col2_values SYS.ODCIVARCHAR2LIST;
BEGIN
select col1, col2
BULK COLLECT INTO col1_values, col2_values
from Table1
Where Col3 in ( a001, a002, a003, a004 );
END;
/
Below is my problem and desired solution.
Query1:
Select colnames from table1;
Query1 Result:
col1
col2
col3
col4
Query2:
Select a1.*
from table2 a1;
-- should translate to
select a1.col1, a1.col2, a1.col3, a1.col4 from table2 a1;
My first query will give the list of column names, I need to replace the .* with those column names in my second query. How can I achieve this?
You are looking for dynamic SQL. The idea is to generate the query string from the results of a SQL query. You can then run it with execute immediate.
In your use case, that would look like:
declare
p_sql varchar2(100);
begin
select
'select '
|| listagg('a1.' || colnames, ', ') within group(order by colnames)
|| ' from table2 a1'
into p_sql
from table1;
dbms_output.put_line('sql: ' || p_sql); -- debug
execute immediate p_sql; -- execute
end;
/
For your sample data, this generates:
dbms_output:
sql: select a1.col1, a1.col2, a1.col3, a1.col4 from table2 a1
I need to count the number of null values of all the columns in a table in Oracle.
For instance, I execute the following statements to create a table TEST and insert data.
CREATE TABLE TEST
( A VARCHAR2(20 BYTE),
B VARCHAR2(20 BYTE),
C VARCHAR2(20 BYTE)
);
Insert into TEST (A) values ('a');
Insert into TEST (B) values ('b');
Insert into TEST (C) values ('c');
Now, I write the following code to compute the number of null values in the table TEST:
declare
cnt number :=0;
temp number :=0;
begin
for r in ( select column_name, data_type
from user_tab_columns
where table_name = upper('test')
order by column_id )
loop
if r.data_type <> 'NOT NULL' then
select count(*) into temp FROM TEST where r.column_name IS NULL;
cnt := cnt + temp;
END IF;
end loop;
dbms_output.put_line('Total: '||cnt);
end;
/
It returns 0, when the expected value is 6.
Where is the error?
Thanks in advance.
Counting NULLs for each column
In order to count NULL values for all columns of a table T you could run
SELECT COUNT(*) - COUNT(col1) col1_nulls
, COUNT(*) - COUNT(col2) col2_nulls
,..
, COUNT(*) - COUNT(colN) colN_nulls
, COUNT(*) total_rows
FROM T
/
Where col1, col2, .., colN should be replaced with actual names of columns of T table.
Aggregate functions -like COUNT()- ignore NULL values, so COUNT(*) - COUNT(col) will give you how many nulls for each column.
Summarize all NULLs of a table
If you want to know how many fields are NULL, I mean every NULL of every record you can
WITH d as (
SELECT COUNT(*) - COUNT(col1) col1_nulls
, COUNT(*) - COUNT(col2) col2_nulls
,..
, COUNT(*) - COUNT(colN) colN_nulls
, COUNT(*) total_rows
FROM T
) SELECT col1_nulls + col1_nulls +..+ colN_null
FROM d
/
Summarize all NULLs of a table (using Oracle dictionary tables)
Following is an improvement in which you need to now nothing but table name and it is very easy to code a function based on it
DECLARE
T VARCHAR2(64) := '<YOUR TABLE NAME>';
expr VARCHAR2(32767);
q INTEGER;
BEGIN
SELECT 'SELECT /*+FULL(T) PARALLEL(T)*/' || COUNT(*) || ' * COUNT(*) OVER () - ' || LISTAGG('COUNT(' || COLUMN_NAME || ')', ' + ') WITHIN GROUP (ORDER BY COLUMN_ID) || ' FROM ' || T
INTO expr
FROM USER_TAB_COLUMNS
WHERE TABLE_NAME = T;
-- This line is for debugging purposes only
DBMS_OUTPUT.PUT_LINE(expr);
EXECUTE IMMEDIATE expr INTO q;
DBMS_OUTPUT.PUT_LINE(q);
END;
/
Due to calculation implies a full table scan, code produced in expr variable was optimized for parallel running.
User defined function null_fields
Function version, also includes an optional parameter to be able to run on other schemas.
CREATE OR REPLACE FUNCTION null_fields(table_name IN VARCHAR2, owner IN VARCHAR2 DEFAULT USER)
RETURN INTEGER IS
T VARCHAR2(64) := UPPER(table_name);
o VARCHAR2(64) := UPPER(owner);
expr VARCHAR2(32767);
q INTEGER;
BEGIN
SELECT 'SELECT /*+FULL(T) PARALLEL(T)*/' || COUNT(*) || ' * COUNT(*) OVER () - ' || listagg('COUNT(' || column_name || ')', ' + ') WITHIN GROUP (ORDER BY column_id) || ' FROM ' || o || '.' || T || ' t'
INTO expr
FROM all_tab_columns
WHERE table_name = T;
EXECUTE IMMEDIATE expr INTO q;
RETURN q;
END;
/
-- Usage 1
SELECT null_fields('<your table name>') FROM dual
/
-- Usage 2
SELECT null_fields('<your table name>', '<table owner>') FROM dual
/
Thank you #Lord Peter :
The below PL/SQL script works
declare
cnt number :=0;
temp number :=0;
begin
for r in ( select column_name, nullable
from user_tab_columns
where table_name = upper('test')
order by column_id )
loop
if r.nullable = 'Y' then
EXECUTE IMMEDIATE 'SELECT count(*) FROM test where '|| r.column_name ||' IS NULL' into temp ;
cnt := cnt + temp;
END IF;
end loop;
dbms_output.put_line('Total: '||cnt);
end;
/
The table name test may be replaced the name of table of your interest.
I hope this solution is useful!
The dynamic SQL you execute (this is the string used in EXECUTE IMMEDIATE) should be
select sum(
decode(a,null,1,0)
+decode(b,null,1,0)
+decode(c,null,1,0)
) nullcols
from test;
Where each summand corresponds to a NOT NULL column.
Here only one table scan is necessary to get the result.
Use the data dictionary to find the number of NULL values almost instantly:
select sum(num_nulls) sum_num_nulls
from all_tab_columns
where owner = user
and table_name = 'TEST';
SUM_NUM_NULLS
-------------
6
The values will only be correct if optimizer statistics were gathered recently and if they were gathered with the default value for the sample size.
Those may seem like large caveats but it's worth becoming familiar with your database's statistics gathering process anyway. If your database is not automatically gathering statistics or if your database is not using the default sample size those are likely huge problems you need to be aware of.
To manually gather stats for a specific table a statement like this will work:
begin
dbms_stats.gather_table_stats(user, 'TEST');
end;
/
select COUNT(1) TOTAL from table where COLUMN is NULL;
How can we get null and non-null counts of all columns of a Table in Vertica? Table can have n number of columns and for each column we need to get count of nulls and non-nulls values of that table.
For Example.
Below Table has two columns
column1 Column2
1 abc
pqr
3
asd
5
If its a specific column then we can check like
SELECT COUNT(*) FROM table where column1 is null;
SELECT COUNT(*) FROM table where column1 is not null;
Same query for column2
I checked system tables like projection_storage and others but I cant figure out a generic query which gives details by hard coding only TABLE NAME in the query.
Hello #user2452689: Here is a dynamically generated VSQL statement which meets your requirement of counting nulls & not nulls in N columns. Notice that this writes a temporary SQL file out to your working directory, and then execute it via the \i command. You only need to change the first two variables per table. Hope this helps - good luck! :-D
--CHANGE SCHEMA AND TABLE PARAMETERS ONLY:
\set table_schema '\'public\''
\set table_name '\'dim_promotion\''
---------
\o temp_sql_file
\pset tuples_only
select e'select \'' || :table_schema || e'\.' || :table_name || e'\' as table_source' as txt
union all
select * from (
select
', sum(case when ' || column_name || ' is not null then 1 else 0 end) as ' || column_name || '_NOT_NULL
, sum(case when ' || column_name || ' is null then 1 else 0 end) as ' || column_name || '_NULL' as txt
from columns
where table_schema = :table_schema
and table_name = :table_name
order by ordinal_position
) x
union all
select ' from ' || :table_schema || e'.' || :table_name || ';' as txt ;
\o
\pset tuples_only
\i temp_sql_file
You can use:
select count(*) as cnt,
count(column1) as cnt_column1,
count(column2) as cnt_column2
from t;
count() with a column name or expression counts the number of non-NULL values in the column/expression.
(Obviously, the number of NULL values is cnt - cnt_columnX.)
select column1_not_null
,column2_not_null
,column3_not_null
,cnt - column1_not_null as column1_null
,cnt - column2_not_null as column2_null
,cnt - column3_not_null as column3_null
from (select count(*) as cnt
,count (column1) as column1_not_null
,count (column2) as column2_not_null
,count (column3) as column3_not_null
from mytable
) t
For various reasons the organisation I work for has data stored on both Oracle and MS SQL server databases. We are moving some static historical data over and I have to check that the data has been moved properly.
The Query below checks the data in SQL server and produces a table listing counts of all the values in each column of the table.
Due to formatting differences in Oracle I will need to group by two other columns Year and Iteration_count . I have not been able to get a loop through of all columns in a table working in Oracle as my experience is pretty much limited to SQL Server
DECLARE #SQL NVARCHAR(MAX) = ''
SELECT #SQL = STUFF((SELECT ' UNION SELECT ''' + name
+ ''' AS [Column], '
+ 'CAST(' + QUOTENAME(Name)
+ ' AS NVARCHAR(MAX)) AS [ColumnValue], COUNT(*) AS [Count] FROM '
+'dbo.HES_APC_ACP_9798'
+' where (NUMACP IS NOT NULL AND NOT (NUMACP = 0) ) '
+' GROUP BY ' + QUOTENAME(Name)
--+'Order By [Column],[ColumnValue]'
FROM sys.columns
WHERE object_id = Object_id('dbo.HES_APC_ACP_9798' )
FOR XML PATH ('')), 1, 7, '');
EXECUTE sp_executesql #SQL;
This loop on user_tab_columns should help:
declare
v_table varchar2(30) := 'TEST';
v_sql varchar2(32767);
begin
for r in (select column_name name from user_tab_cols
where table_name=v_table order by column_id)
loop
v_sql := v_sql||' union all select '''||r.name||''' col_name, to_char('
||r.name||') col_value, count(1) cnt from '||v_table
||' group by '||r.name||chr(13);
end loop;
v_sql := ltrim(v_sql, ' union all ');
dbms_output.put_line(v_sql);
end;
Test table:
create table test (col1 varchar2(10), col2 number(5), col3 date);
insert into test values ('ABC', 1, null);
insert into test values ('DEF', 1, date '2015-06-18');
Executing first PLSQL block outputs:
select 'COL1' col_name, to_char(COL1) col_value, count(1) cnt from TEST group by COL1
union all select 'COL2' col_name, to_char(COL2) col_value, count(1) cnt from TEST group by COL2
union all select 'COL3' col_name, to_char(COL3) col_value, count(1) cnt from TEST group by COL3
Output of this query:
COL_NAME COL_VALUE CNT
-------- ------------ ----------
COL1 DEF 1
COL1 ABC 1
COL2 1 2
COL3 1
COL3 15/06/18 1
Use all_tab_cols and add filter for owner if you read data from other schema. You can also run generated query using execute immediate statement.