Adding a element to an existing struct in a Spark dataframe - dataframe

Question

I have a PySpark dataframe and the schema looks like this:
root
|-- useragent: string (nullable = true)
|-- properties: struct (nullable = true)
| |-- browser: string (nullable = true)
| |-- device: string (nullable = true)
I am using udf + withColumn to extract more info out of the useragent. But I am only able to store any additional property I am creating in a new column. Is there a way I can append it to the struct itself?
I've changed the schema but it doesn't transfer all the data I needed from the other properties that already exist
df = df.schema['properties'].dataType.add(StructField('type', StringType()))
Would it make more sense to convert this to a RDD?

Answer 1

You will not get any data using this piece of code, since you are not selecting any data.
What you are doing in this operation is nothing but accessing the Schema property, modifying it in some way and assigning it to a variable df (this is not the modified dataframe you are expecting) but just a StructType object (There seems to be some syntax error there, please check)
What you should do is to modify the dataframe through select and suitable transformation functions. You may considering creating a new column of type StructField with the required structure and add it as a new column using withColumn function

Answer 2

withField can be used:
F.col("properties").withField("type", extract_function)
Full example:
from pyspark.sql import functions as F
df = spark.createDataFrame(
[("type=type1,mode=5", ("Chrome", "PC1"))],
"useragent string, properties struct<browser:string,device:string>"
)
df.printSchema()
# root
# |-- useragent: string (nullable = true)
# |-- properties: struct (nullable = true)
# | |-- browser: string (nullable = true)
# | |-- device: string (nullable = true)
df.show()
# +-----------------+-------------+
# | useragent| properties|
# +-----------------+-------------+
# |type=type1,mode=5|{Chrome, PC1}|
# +-----------------+-------------+
df = df.withColumn("properties", F.col("properties").withField(
"type",
F.regexp_extract("useragent", r"type=(\w+)", 1)
))
df.printSchema()
# root
# |-- useragent: string (nullable = true)
# |-- properties: struct (nullable = true)
# | |-- browser: string (nullable = true)
# | |-- device: string (nullable = true)
# | |-- type: string (nullable = true)
df.show()
# +-----------------+--------------------+
# | useragent| properties|
# +-----------------+--------------------+
# |type=type1,mode=5|{Chrome, PC1, type1}|
# +-----------------+--------------------+