How to pass columns as a JSON record to the API method using Pyspark? - api

Question

I have a dataset where I segregate the records in 4K batch for a DF and then try to hit the API. The first part is working fine. The problem occurs when I try to hit the API. I want to hit the API in the form of JSON records. Therefore, I am passing a struct of values converted as JSON to the call_to_cust_bulk_api method. Inside that method, I pass to a custRequestBody and then send that data to the API. The code is below:
def call_to_cust_bulk_api(url, payload):
print("Calling Bulk API")
try:
print(payload)
token = get_token(tokenUrl, tokenBody)
custRequestBody = {
"Token": token,
"CustomerName": "",
"Object": "",
"Data": payload
}
# TODO: write code...
headers = {'content-type': 'application/json'}
print(":::::::jsn load::::")
response = requests.post(url, data=custRequestBody, headers=headers)
print(":::Response::::", response)
data = response.json()
return data
except Exception as e:
print('ExceptionInPushing ' + str(e))
df = spark.read.option("header", "true").csv(
".csv",
sep="~")
df = df.withColumn("uniqueID", lit("1"))
df = df.withColumn("row_num", row_number().over(
Window.partitionBy(col("uniqueID")).orderBy(col("uniqueID"))
))
total_count = df.count()
i = 1
while i < total_count:
rangeNum = i + 3999
print("Range Num:::")
print(rangeNum)
df1 = df.filter((col("row_num") >= i) & (col("row_num") <= rangeNum))
finalDF = df1.drop("row_num", "edl_timestamp", "uniqueID")
colsListToBePassed = finalDF.columns
print("finalDF count:::", finalDF.count())
finalDF = finalDF.repartition(finalDF.rdd.getNumPartitions()).withColumn("status_for_batch",
call_to_cust_bulk_api(policyUrl,
to_json(
struct(*colsListToBePassed))))
It gives me the below error:
Traceback (most recent call last):
File "/home/lumiq/IdeaProjects/pyspark_python_test/com/apitest3.py", line 116, in <module>
finalDF = finalDF.repartition(finalDF.rdd.getNumPartitions()).withColumn("status_for_batch",
File "/home/lumiq/Downloads/spark-3.2.1-bin-hadoop3.2/python/pyspark/sql/dataframe.py", line 2477, in withColumn
raise TypeError("col should be Column")
TypeError: col should be Column
So, I want to track the response as a new column and write this data frame later.
So, where am I going wrong?