How to replace nulls with zeros in pivot query sql for fact table in Databricks - apache-spark-sql

Question

I see lots of solutions for how to do this where there is a column being queried including the following...
how to Replace null with zero in pivot SQL query
Oracle 11g SQL - Replacing NULLS with zero where query has PIVOT
Replacing null values in dynamic pivot sql query
etc.,etc.,etc.,
But how do you replace the nulls in a pivot query when your are creating a fact table for the existence of a condition.
For example, in Databricks:
How do I replace the nulls for the following
Setup
drop table if exists patient_dx;
create table patient_dx (patient_id string, dx string);
insert into patient_dx values
('Bob', 'cough'),
('Donna', 'cough'),
('Jerry', 'cough'),
('Bob', 'feaver'),
('Donna', 'head ache')
;
Query:
select * from (
select
patient_id,
dx,
cast (1 as int) cnt
from
patient_dx
)
pivot (
max(cnt)
for dx in ('cough','feaver','head ache')
)
;
Result
I've tried several permutations of:
cast(0 + cast(coalesce(sum(coalesce(cnt,0)),0) as int) as int) as cnt
To no avail

Answer 1

You have to use coalesce or NOT NULL to substitute null values in select query.
Check below if it helps:
Try this:
spark.sql("""
select
patient_id,
CASE
when cough is NOT NULL THEN cough
else 0
END as cough,
CASE
when feaver is NOT NULL THEN feaver
else 0
END as feaver,
CASE
when `head ache` is NOT NULL THEN `head ache`
else 0
END as `head ache`
from (
select * from patient
)
PIVOT(
Count(dx)
for dx in ('cough','feaver','head ache')
)
;
""").show()
The output will be:
patient_id
cough
feaver
head ache
Donna
1
0
1
Jerry
1
0
0
Bob
1
1
0
if you want it to be dynamic
dist=spark.sql("select collect_set(dx) from patient;").toPandas()
val=spark.sql("""
select
patient_id,
coalesce(cough,0) as `cough`,
coalesce(feaver,0) as `feaver`,
coalesce(`head ache`,0) as `head ache`
from (
select * from patient
)
PIVOT(
Count(dx)
for dx in """
+
str(tuple(map(tuple, *dist.values))[0])
+
"""
)
;
""")