Multiprocessing on dataset from pyspark returns JVM error - pandas

Question

I need to run some clustering algorithms in parallel in Jupyter notebook. The clustering function I want to parallel works when doing multithreading or when run individually. However, it returns
raise Py4JError("{0} does not exist in the JVM".format(name))
when I try multiprocessing. I don't have much experience with multiprocessing, what could I be doing wrong?
Code for clustering:
def clustering(ID, df):
pandas_df = df.select("row", "features", "type") \
.where(df.type == ID).toPandas()
print("process " + str(ID) + ": preparing data for clustering")
feature_series = pandas_df["features"].apply(lambda x: x.toArray())
objs = [pandas_df, pd.DataFrame(feature_series.tolist())]
t_df = pd.concat(objs, axis=1)
print("process " + str(ID) + ": initiating clustering")
c= #clustering algo here
print("process " + str(ID) + " DONE!")
return
Code for multiprocessing:
import multiprocessing as mp
k = 4
if __name__ == '__main__':
pl = []
for i in range(0,k):
print("sending process:", i)
process = mp.Process(target=clustering, args=(i, df))
jobs.append(process)
process.start()
for process in pl:
print("waiting for join from process")
process.join()

Answer 1

Error was caused by the subprocesses not being able to access the same memory(in which the pyspark dataframe resided).
Solved by partitioning the dataset first by putting the access to the pyspark dataframe in another function like so:
pandas_df = df.select("row", "features", "type") \
.where(df.type == ID).toPandas()
And then running the clustering on the separated Pandas dataframes.