How to split the names into different column - apache-spark-sql

Question

How to split the full name into different columns in pyspark.
input CSV:
Name,Marks
Sam Kumar Timberlake,83
Theo Kumar Biber,82
Tom Kumar Perry,86
Xavier Kumar Cruse,87
Output Csv should be :
FirstName,MiddleName,LastName,Marks
Sam,Kumar,Timberlake,83
Theo,Kumar,Biber,82
Tom,Kumar,Perry,86
Xavier,Kumar,Cruse,87

Answer 1

I am sure there is a better way, but the longer way is re-instate. Meaning, do the work. I created double of the names and just manually cleaned the data into first middle names and last names. I don't think there is any machine language that can tell you the person has two first names and one middle name unless the person used a dash for two first names and two last names (born and married into last names) and use common sense for last names and be ready for mistakes. Gotta do it manually, unless, again.. you are certain because you called them up and know for sure.
The mathematical way would be separating last name from the rest. It is like calling someone by their first name John when they go by their middle name Gary. Mistakes are inevitable as long the person you address understands it is legally them. Not sure if it all makes sense.

Answer 2

This should work in your specific case:
import pyspark.sql.functions as F
df = df.withColumn(
"arr", F.split(F.col("Name"), " ")
)
df = (
df
.withColumn('FirstName', F.arr.getItem(0))
.withColumn('MiddleName', F.arr.getItem(1))
.withColumn('LastName', F.arr.getItem(2))
)
If you want to include the case when someone has several middle names:
df = (
df
.withColumn('FirstName', df.arr.getItem(0))
.withColumn('LastName', df.arr[F.size(df.arr)-1])
)
df = df.withColumn(
'MiddleName',
F.trim(F.expr("substring(Name, length(FirstName)+1, length(Name)-length(LastName)-length(FirstName))"))
)