Pyspark Dataframe Merge Rows by eliminating null values - dataframe

Question

i have a Pyspark Data Frame like this one
+-----------+-------+----------+-------+-------+---------+
| ID_PRODUCT| VALUE | TIMESTAMP| SPEED | CODE | FIRMWARE|
+-----------+-------+----------+-------+-------+---------+
| 3| 1| null| 124,21| null| null|
| 5| 2| null| 124,23| null| null|
| 5| 2| null| 124,26| null| null|
| 6| 4| null| 124,24| null| null|
| 3| 1| null| null| 6764| null|
| 5| 2| null| null| 6772| null|
| 5| 2| null| null| 6782| null|
| 6| 4| null| null| 6932| null|
| 3| 1| null| null| null| 1|
| 5| 2| null| null| null| 1|
| 5| 2| null| null| null| 1|
| 6| 4| null| null| null| 1|
| 3| 1| 17:18:04| null| null| null|
| 5| 2| 18:22:40| null| null| null|
| 5| 2| 18:25:29| null| null| null|
| 6| 4| 18:32:18| null| null| null|
+-----------+-------+----------+-------+-------+---------+
and i want to merge the columns of it, it should look like (for example):
+-----------+-------+----------+-------+-------+---------+
| ID_PRODUCT| VALUE | TIMESTAMP| SPEED | CODE | FIRMWARE|
+-----------+-------+----------+-------+-------+---------+
| 3| 1| 17:18:04| 124,21| 6764| 1|
| 5| 2| 18:22:40| 124,23| 6772| 1|
| 5| 2| 18:25:29| 124,26| 6782| 1|
| 6| 4| 18:32:18| 124,24| 6932| 1|
+-----------+-------+----------+-------+-------+---------+
I tried to use:
df = df.groupBy('id').agg(*[f.first(x,ignorenulls=True) for x in df.columns])
however, this is just giving me just the first value of the column and i need all the records. Because to one ID i have different registered Timestamps and different registered values, which im now loosing.
Thanks for the advice

Answer 1

I'm not sure if this is what you wanted, but essentially you can do a collect_list for each id and column, and explode all resulting lists. In this way, you can have multiple entries per id.
from functools import reduce
import pyspark.sql.functions as F
df2 = reduce(
lambda x, y: x.withColumn(y, F.explode_outer(y)),
df.columns[2:],
df.groupBy('id_product', 'value').agg(*[F.collect_list(c).alias(c) for c in df.columns[2:]])
).distinct()