How to transform pyspark dataframe 1x9 to 3x3 - dataframe

Question

Im using pyspark dataframe.
I have a df which is 1x9
example
temp = spark.read.option("sep","\n").csv("temp.txt")
temp :
sam
11
newyork
john
13
boston
eric
22
texas
without using Pandas library, How can I transform this to 3x3 dataframe with columns name,age,city ?
like this :
name,age,city
sam,11,newyork
john,13,boston

Answer 1

I would read the file as an rdd to take advantage of zipWithIndex to add an index to your data.
rdd = sc.textFile("temp.txt")
We can now use truncating division to create an index with which to group records together. Use this new index as the key for the rdd. The corresponding values will be a tuple of the header, which can be computed using the modulus, and the actual value. (Note the index returned by zipWithIndex will be at the end of the record, which is why we use row[1] for the division/mod.)
Next use reduceByKey to add the value tuples together. This will give you a tuple of keys and values (in sequence). Use map to turn that into a Row (to keep column headers, etc).
Finally use toDF() to convert to a DataFrame. You can use select(header) to get the columns in the desired order.
from operator import add
from pyspark.sql import Row
header = ["name", "age", "city"]
df = rdd.zipWithIndex()\
.map(lambda row: (row[1]//3, (header[row[1]%3], row[0])))\
.reduceByKey(add)\
.map(lambda row: Row(**dict(zip(row[1][::2], row[1][1::2]))))\
.toDF()\
.select(header)
df.show()
#+----+---+-------+
#|name|age| city|
#+----+---+-------+
#| sam| 11|newyork|
#|eric| 22| texas|
#|john| 13| boston|
#+----+---+-------+