pyspark: isIN and isNOT IN replcaement with another df column - dataframe

Question

i'm trying to filter dataframe in pyspark using "isin"
also tried another way of filtering.
unable to get the correct result.
getting error of Spark Array literal.
can anyone help
One way:
df1.select("COL1").distinct().show()
df2.select(('col1').isin(df1.select("COL1").distinct()))
-------
Second way :
uniquelist=df1.select("COL1").distinct().collect()
df2.filter(F.col('col1').contains(uniqueVIN)).show()
can anyone help me solve the error :
An error occurred while calling z:org.apache.spark.sql.functions.lit.
I also have to perform a "is not in"
data_array = np.array(df_list.select("f_col").collect())
df_filtered = df_2.filter(~df_2["colname"].isin([data_array]))

Answer 1

collect() returns a list of Row objects, you need to get the values from the rows before passing it to isin column method:
unique_list = [r["COL1"] for r in df1.select("COL1").distinct().collect()]
df2.filter(F.col('col1').isin(unique_list)).show()
However, you should use join for this:
Use left_semi to get row from df2 with corresponding rows from df1:
df2.join(df1, df1["COL1"] == df2["col1"], "left_semi").show()
And left_anti to get rows from df2 that have no corresponding values in df1:
df2.join(df1, df1["COL1"] == df2["col1"], "left_anti").show()