I have a table (WORKLOG) in an Oracle SQL database which has a VARCHAR description field of 250 characters. On a separate table (LONGDESCRIPTION) there is a CLOB field which holds a long description. My goal is to build a query which returns the first 4000 characters of the concatenation of the short and long descriptions.
I initially tried this with SUBSTR functions, but that didn't work so well with the CLOB: it was overflowing the VARCHAR(4000) field I'd set up for it (ORA-64203: Destination buffer too small to hold CLOB data after character set conversion.). After searching on here I tried using DBMS_LOB.SUBSTR, but this didn't seem to work either: I got the same 64203, or ORA-22831: Offset or offset+amount does not land on character boundary. I have tried a few ways to fix it, using slightly different functions, but each runs into trouble, and I'm having no luck working out a way to avoid cutting a character but also ensuring I don't overflow the 4000 limit.
Here's the code I'm using. The idea is that it should use the short description if there is no long description, use a truncated long description if there is no short description or if the short description is the same as the start of the long description, and otherwise concatenate the two. The REGEXP functions are to try and remove HTML formatting.
SELECT
WORKLOG.WORKLOGID as Worklog,
WORKLOG.DESCRIPTION as Description,
CASE
WHEN WORKLOG.DESCRIPTION is null AND LENGTH(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>')) > 4000
THEN CAST(SUBSTR(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>'), 1, 3996) as VARCHAR(3996)) || '...'
WHEN WORKLOG.DESCRIPTION is null
THEN CAST(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>') as VARCHAR(4000))
WHEN LONGDESCRIPTION.LDKEY is null
THEN WORKLOG.DESCRIPTION
WHEN CAST(SUBSTR(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>'), 1, LENGTH(WORKLOG.DESCRIPTION)) as VARCHAR(4000)) = WORKLOG.DESCRIPTION AND LENGTH(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>')) > 4000
THEN CAST(SUBSTR(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>'), 1, 3996) as VARCHAR(3996)) || '...'
WHEN CAST(SUBSTR(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>'), 1, LENGTH(WORKLOG.DESCRIPTION)) as VARCHAR(4000)) = WORKLOG.DESCRIPTION
THEN CAST(SUBSTR(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>'), 1, 4000) as VARCHAR(4000))
WHEN LENGTH(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>')) > 3732
THEN WORKLOG.DESCRIPTION || ' // ' || CAST(SUBSTR(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>'), 1, 3732) as VARCHAR(3732)) || '...'
ELSE WORKLOG.DESCRIPTION || ' // ' || CAST(SUBSTR(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>'), 1, 3732) as VARCHAR(3735))
END as Description_Full
FROM
Maximo.WORKLOG
LEFT JOIN
Maximo.LONGDESCRIPTION
ON LONGDESCRIPTION.LDOWNERTABLE = 'WORKLOG'
AND LONGDESCRIPTION.LDOWNERCOL = 'DESCRIPTION'
AND LONGDESCRIPTION.LDKEY = WORKLOG.WORKLOGID
AND LENGTH(REGEXP_REPLACE(LONGDESCRIPTION.LDTEXT, '<.*?>')) > 0
Examples where the description field is 2 characters long and the maximum is 10:
Description
Long description
Expected result
ABC
ABC
ABCDEFGHIJKL
ABCDEFGHIJ
AB
AB
AB
ABCDE
ABCDE
AB
ABCDEFGHIJKL
ABCDEFGHIJ
AB
XY
AB // XY
AB
XYZABCDEF
AB // XYZA
Any help would be greatly appreciated.
Assuming that your CLOB contains well-formed XML or XHTML then you should use a proper XML parser (and not regular expressions) and can use:
SELECT w.WORKLOGID as Worklog,
w.DESCRIPTION as Description,
SUBSTR(
CASE
WHEN w.description IS NOT NULL AND xml.text LIKE w.description || '%'
THEN xml.text
WHEN w.description IS NOT NULL AND xml.text IS NOT NULL
THEN TO_CLOB(w.DESCRIPTION) || '//' || xml.text
WHEN w.description IS NOT NULL
THEN TO_CLOB(w.DESCRIPTION)
WHEN xml.text IS NOT NULL
THEN xml.text
END,
1,
4000
) AS Description_Full
FROM /*Maximo.*/WORKLOG w
LEFT JOIN (
/*Maximo.*/LONGDESCRIPTION ld
CROSS APPLY XMLTABLE(
'/HTML/BODY'
PASSING XMLTYPE(
COALESCE(ld.LONGDESCRIPTION, TO_CLOB('<HTML></HTML>') )
)
COLUMNS
text CLOB PATH '.'
) xml
)
ON ld.LDOWNERTABLE = 'WORKLOG'
AND ld.LDOWNERCOL = 'DESCRIPTION'
AND ld.LDKEY = w.WORKLOGID
Which, for the sample data:
CREATE TABLE worklog (
worklogid NUMBER,
description VARCHAR2(250)
);
CREATE TABLE longdescription (
LDOWNERTABLE VARCHAR2(30),
LDOWNERCOL VARCHAR2(30),
LDKEY NUMBER,
LONGDESCRIPTION CLOB
);
INSERT INTO worklog (worklogid, description)
SELECT 1, 'Value1' FROM DUAL UNION ALL
SELECT 2, 'Value2' FROM DUAL UNION ALL
SELECT 3, NULL FROM DUAL UNION ALL
SELECT 4, NULL FROM DUAL;
INSERT INTO longdescription
SELECT 'WORKLOG', 'DESCRIPTION', 1, '<HTML><BODY>Test</BODY></HTML>' FROM DUAL UNION ALL
SELECT 'WORKLOG', 'DESCRIPTION', 2, NULL FROM DUAL UNION ALL
SELECT 'WORKLOG', 'DESCRIPTION', 3, '<HTML><BODY>Test</BODY></HTML>' FROM DUAL UNION ALL
SELECT 'WORKLOG', 'DESCRIPTION', 4, NULL FROM DUAL;
Outputs:
WORKLOG
DESCRIPTION
DESCRIPTION_FULL
1
Value1
Value1//Test
3
null
Test
2
Value2
Value2
4
null
null
db<>fiddle here
I was trying to fetch data with condition "get data only if string contains the lower char in the 4th and 5th position".
Below are the conditions:
and ascii(substr(RAP01.CRDVER,4,1)) between 97 and 122
and ascii(substr(RAP01.CRDVER,5,1)) between 97 and 122;
Table name RAP01 and column name CRDVER.
But It is not fetching all the required data.
Can this approach is correct?
How about this? The key is line 8 - query will return strings whose 4th and 5th character (substr(col, 4, 2))is a lowercase letter ([a-z]) (c means that the search is case sensitive).
SQL> with test (col) as
2 (select '12Bcfx23' from dual union all
3 select '123456' from dual union all
4 select 'ABCDEFGH' from dual
5 )
6 select col, substr(col, 4, 2) sub
7 from test
8 where regexp_like(substr(col, 4, 2), '[a-z]', 'c');
COL SU
-------- --
12Bcfx23 cf
SQL>
Are lower case a subset of ASCII? NO
Can I use ASCII to identify lower case? NO
How can I know if a string has lower case? It has lower case if UPPER of string is not the same as just the string.
Sample:
create table t1 ( a varchar(100) )\\
insert into t1 (a) values ( 'A' )\\
insert into t1 (a) values ( 'a' )\\
insert into t1 (a) values ( 'å' )\\
insert into t1 (a) values ( 'æ' )\\
select (case when a = upper( a ) <-- compare with upper
then a || ' Is Upper'
else a || ' Has lower' end) as r_upper,
(case when not regexp_like(a, '[a-z]', 'c') <-- using ascii
then a || ' Is Upper'
else a || ' Has lower' end) as r_reg
from t1\\
R_UPPER | R_REG
-------------------------
A Is Upper | A Is Upper
a Has lower | a Has lower
å Has lower | å Is Upper (¹)
æ Has lower | æ Is Upper (¹)
(¹) Wrong result using ASCII.
I have a Postgres table with a string column carrying numeric values. I need to convert these strings to numbers for math, but I need both NULL values as well as empty strings to be interpreted as 0.
I can convert empty strings into null values:
# select nullif('','');
nullif
--------
(1 row)
And I can convert null values into a 0:
# select coalesce(NULL,0);
coalesce
----------
0
(1 row)
And I can convert strings into numbers:
# select cast('3' as float);
float8
--------
3
(1 row)
But when I try to combine these techniques, I get errors:
# select cast( nullif( coalesce('',0), '') as float);
ERROR: invalid input syntax for integer: ""
LINE 1: select cast( nullif( coalesce('',0), '') as float);
# select coalesce(nullif('3',''),4) as hi;
ERROR: COALESCE types text and integer cannot be matched
LINE 1: select coalesce(nullif('3',''),4) as hi;
What am I doing wrong?
The types of values need to be consistent; coalescing the empty string to a 0 means that you cannot then compare it to null in the nullif. So either of these works:
# create table tests (orig varchar);
CREATE TABLE
# insert into tests (orig) values ('1'), (''), (NULL), ('0');
INSERT 0 4
# select orig, cast(coalesce(nullif(orig,''),'0') as float) as result from tests;
orig | result
------+--------
1 | 1
| 0
| 0
0 | 0
(4 rows)
# select orig, coalesce(cast(nullif(orig,'') as float),0) as result from tests;
orig | result
------+--------
1 | 1
| 0
| 0
0 | 0
(4 rows)
You could also use
cast(
case
when coalesce(orig, '') = '' then '0'
else orig
end
as float
)
You could also unwrap that a bit since you're being fairly verbose anyway:
cast(
case
when orig is null then '0'
when orig = '' then '0'
else orig
end
as float
)
or you could put the cast inside the CASE:
case
when coalesce(orig, '') = '' then 0.0
else cast(orig as float)
end
A CASE makes it a bit easier to account for any other special conditions, this also seems like a clearer expression of the logic IMO. OTOH, personal taste and all that.
Actually, you can cast NULL to int, you just can't cast an empty string to int. Assuming you want NULL in the new column if data1 contains an empty string or NULL, you can do something like this:
UPDATE table SET data2 = cast(nullif(data1, '') AS int);
or
UPDATE table SET data2 = nullif(data1, '')::int;
Reference
Check if you query parameter is empty (accepts null, empty string or value):
SELECT CAST(TO_JSON(NULLIF(:myParameter, NULL)) AS VARCHAR) IS NULL OR
CAST(TO_JSON(NULLIF(:myParameter, NULL)) AS VARCHAR) IN ('""');
I was wondering about the possibility to count the null columns of row in SQL, I have a table Customer that has nullable values, simply I want a query that return an int of the number of null columns for certain row(certain customer).
This method assigns a 1 or 0 for null columns, and adds them all together. Hopefully you don't have too many nullable columns to add up here...
SELECT
((CASE WHEN col1 IS NULL THEN 1 ELSE 0 END)
+ (CASE WHEN col2 IS NULL THEN 1 ELSE 0 END)
+ (CASE WHEN col3 IS NULL THEN 1 ELSE 0 END)
...
...
+ (CASE WHEN col10 IS NULL THEN 1 ELSE 0 END)) AS sum_of_nulls
FROM table
WHERE Customer=some_cust_id
Note, you can also do this perhaps a little more syntactically cleanly with IF() if your RDBMS supports it.
SELECT
(IF(col1 IS NULL, 1, 0)
+ IF(col2 IS NULL, 1, 0)
+ IF(col3 IS NULL, 1, 0)
...
...
+ IF(col10 IS NULL, 1, 0)) AS sum_of_nulls
FROM table
WHERE Customer=some_cust_id
I tested this pattern against a table and it appears to work properly.
My answer builds on Michael Berkowski's answer, but to avoid having to type out hundreds of column names, what I did was this:
Step 1: Get a list of all of the columns in your table
SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'myTable';
Step 2: Paste the list in Notepad++ (any editor that supports regular expression replacement will work). Then use this replacement pattern
Search:
^(.*)$
Replace:
\(CASE WHEN \1 IS NULL THEN 1 ELSE 0 END\) +
Step 3: Prepend SELECT identityColumnName, and change the very last + to AS NullCount FROM myTable and optionally add an ORDER BY...
SELECT
identityColumnName,
(CASE WHEN column001 IS NULL THEN 1 ELSE 0 END) +
-- ...
(CASE WHEN column200 IS NULL THEN 1 ELSE 0 END) AS NullCount
FROM
myTable
ORDER BY
NullCount DESC
For ORACLE-DBMS only.
You can use the NVL2 function:
NVL2( string1, value_if_not_null, value_if_null )
Here is a select with a similiar approach as Michael Berkowski suggested:
SELECT (NVL2(col1, 0, 1)
+ NVL2(col2, 0, 1)
+ NVL2(col3, 0, 1)
...
...
+ NVL2(col10, 0, 1)
) AS sum_of_nulls
FROM table
WHERE Customer=some_cust_id
A more generic approach would be to write a PL/SQL-block and use dynamic SQL. You have to build a SELECT string with the NVL2 method from above for every column in the all_tab_columns of a specific table.
Unfortunately, in a standard SQL statement you will have to enter each column you want to test, to test all programatically you could use T-SQL. A word of warning though, ensure you are working with genuine NULLS, you can have blank stored values that the database will not recognise as a true NULL (I know this sounds strange).
You can avoid this by capturing the blank values and the NULLS in a statement like this:
CASE WHEN col1 & '' = '' THEN 1 ELSE 0 END
Or in some databases such as Oracle (not sure if there are any others) you would use:
CASE WHEN col1 || '' = '' THEN 1 ELSE 0 END
You don't state RDBMS. For SQL Server 2008...
SELECT CustomerId,
(SELECT COUNT(*) - COUNT(C)
FROM (VALUES(CAST(Col1 AS SQL_VARIANT)),
(Col2),
/*....*/
(Col9),
(Col10)) T(C)) AS NumberOfNulls
FROM Customer
Depending on what you want to do, and if you ignore mavens, and if you use SQL Server 2012, you could to it another way. .
The total number of candidate columns ("slots") must be known.
1. Select all the known "slots" column by column (they're known).
2. Unpivot that result to get a
table with one row per original column. This works because the null columns don't
unpivot, and you know all the column names.
3. Count(*) the result to get the number of non-nulls;
subtract from that to get your answer.
Like this, for 4 "seats" in a car
select 'empty seats' = 4 - count(*)
from
(
select carId, seat1,seat2,seat3,seat4 from cars where carId = #carId
) carSpec
unpivot (FieldValue FOR seat in ([seat1],[seat2],[seat3],[seat4])) AS results
This is useful if you may need to do more later than just count the number of non-null columns, as it gives you a way to manipulate the columns as a set too.
This will give you the number of columns which are not null. you can apply this appropriately
SELECT ISNULL(COUNT(col1),'') + ISNULL(COUNT(col2),'') +ISNULL(COUNT(col3),'')
FROM TABLENAME
WHERE ID=1
The below script gives you the NULL value count within a row i.e. how many columns do not have values.
{SELECT
*,
(SELECT COUNT(*)
FROM (VALUES (Tab.Col1)
,(Tab.Col2)
,(Tab.Col3)
,(Tab.Col4)) InnerTab(Col)
WHERE Col IS NULL) NullColumnCount
FROM (VALUES(1,2,3,4)
,(NULL,2,NULL,4)
,(1,NULL,NULL,NULL)) Tab(Col1,Col2,Col3,Col4) }
Just to demonstrate I am using an inline table in my example.
Try to cast or convert all column values to a common type it will help you to compare the column of different type.
I haven't tested it yet, but I'd try to do it using a PL\SQL function
CREATE OR REPLACE TYPE ANYARRAY AS TABLE OF ANYDATA
;
CREATE OR REPLACE Function COUNT_NULL
( ARR IN ANYARRAY )
RETURN number
IS
cnumber number ;
BEGIN
for i in 1 .. ARR.count loop
if ARR(i).column_value is null then
cnumber := cnumber + 1;
end if;
end loop;
RETURN cnumber;
EXCEPTION
WHEN OTHERS THEN
raise_application_error
(-20001,'An error was encountered - '
||SQLCODE||' -ERROR- '||SQLERRM);
END
;
Then use it in a select query like this
CREATE TABLE TEST (A NUMBER, B NUMBER, C NUMBER);
INSERT INTO TEST (NULL,NULL,NULL);
INSERT INTO TEST (1 ,NULL,NULL);
INSERT INTO TEST (1 ,2 ,NULL);
INSERT INTO TEST (1 ,2 ,3 );
SELECT ROWNUM,COUNT_NULL(A,B,C) AS NULL_COUNT FROM TEST;
Expected output
ROWNUM | NULL_COUNT
-------+-----------
1 | 3
2 | 2
3 | 1
4 | 0
This is how i tried
CREATE TABLE #temptablelocal (id int NOT NULL, column1 varchar(10) NULL, column2 varchar(10) NULL, column3 varchar(10) NULL, column4 varchar(10) NULL, column5 varchar(10) NULL, column6 varchar(10) NULL);
INSERT INTO #temptablelocal
VALUES (1,
NULL,
'a',
NULL,
'b',
NULL,
'c')
SELECT *
FROM #temptablelocal
WHERE id =1
SELECT count(1) countnull
FROM
(SELECT a.ID,
b.column_title,
column_val = CASE b.column_title
WHEN 'column1' THEN a.column1
WHEN 'column2' THEN a.column2
WHEN 'column3' THEN a.column3
WHEN 'column4' THEN a.column4
WHEN 'column5' THEN a.column5
WHEN 'column6' THEN a.column6
END
FROM
( SELECT id,
column1,
column2,
column3,
column4,
column5,
column6
FROM #temptablelocal
WHERE id =1 ) a
CROSS JOIN
( SELECT 'column1'
UNION ALL SELECT 'column2'
UNION ALL SELECT 'column3'
UNION ALL SELECT 'column4'
UNION ALL SELECT 'column5'
UNION ALL SELECT 'column6' ) b (column_title) ) AS pop WHERE column_val IS NULL
DROP TABLE #temptablelocal
Similary, but dynamically:
drop table if exists myschema.table_with_nulls;
create table myschema.table_with_nulls as
select
n1::integer,
n2::integer,
n3::integer,
n4::integer,
c1::character varying,
c2::character varying,
c3::character varying,
c4::character varying
from
(
values
(1,2,3,4,'a','b','c','d'),
(1,2,3,null,'a','b','c',null),
(1,2,null,null,'a','b',null,null),
(1,null,null,null,'a',null,null,null)
) as test_records(n1, n2, n3, n4, c1, c2, c3, c4);
drop function if exists myschema.count_nulls(varchar,varchar);
create function myschema.count_nulls(schemaname varchar, tablename varchar) returns void as
$BODY$
declare
calc varchar;
sqlstring varchar;
begin
select
array_to_string(array_agg('(' || trim(column_name) || ' is null)::integer'),' + ')
into
calc
from
information_schema.columns
where
table_schema in ('myschema')
and table_name in ('table_with_nulls');
sqlstring = 'create temp view count_nulls as select *, ' || calc || '::integer as count_nulls from myschema.table_with_nulls';
execute sqlstring;
return;
end;
$BODY$ LANGUAGE plpgsql STRICT;
select * from myschema.count_nulls('myschema'::varchar,'table_with_nulls'::varchar);
select
*
from
count_nulls;
Though I see that I didn't finish parametising the function.
My answer builds on Drew Chapin's answer, but with changes to get the result using a single script:
use <add_database_here>;
Declare #val Varchar(MAX);
Select #val = COALESCE(#val + str, str) From
(SELECT
'(CASE WHEN '+COLUMN_NAME+' IS NULL THEN 1 ELSE 0 END) +' str
FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '<add table name here>'
) t1 -- getting column names and adding the case when to replace NULLs for zeros or ones
Select #val = SUBSTRING(#val,1,LEN(#val) - 1) -- removing trailling add sign
Select #val = 'SELECT <add_identity_column_here>, ' + #val + ' AS NullCount FROM <add table name here>' -- adding the 'select' for the column identity, the 'alias' for the null count column, and the 'from'
EXEC (#val) --executing the resulting sql
With ORACLE:
Number_of_columns - json_value( json_array( comma separated list of columns ), '$.size()' ) from your_table
json_array will build an array with only the non null columns and the json_query expression will give you the size of the array
There isn't a straightforward way of doing so like there would be with counting rows. Basically, you have to enumerate all the columns that might be null in one expression.
So for a table with possibly null columns a, b, c, you could do this:
SELECT key_column, COALESCE(a,0) + COALESCE(b,0) + COALESCE(c,0) null_col_count
FROM my_table