I read in data from Snowflake into AWS Glue using spark, which results having a spark dataframe called df. After that I added the following to convert it to a pandas dataframe:
df2 = df.toPandas()
However, this is causing an error in AWS Glue.
Related
If I call sparkSession.createDataFrame() to convert a pandas dataframe into spark dataframe,how will my data be distributed among executors? Are there protential data skewing issues?
I am using an external jar on my spark cluster to load a DataFrame. I am getting an error in attempting to convert this DataFrame to a pandas DataFrame.
The code is as follows :
jvm = spark._jvm
gateway = jvm.com.<external.package.classname>
data_w = gateway.loadData()
df = DataFrame(data_w, spark)
pandas_df = df.toPandas()
The spark dataframe df has valid data. However I am getting an error in the pandas conversion
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py", line 67, in toPandas
'RuntimeConfig' object has no attribute 'sessionLocalTimeZone'
The spark dataframe has 2 date columns. Is there any jconf settings I need to add on my spark context for conversion ?
I can see https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/pandas/conversion.html that there is a usage of sessionLocalTimeZone. Do we need to explicitly set this on spark jconf. I am not setting this locally and it works fine.
I am having issues while trying to convert pandas data frame into spark data frame in Azure. I have done it in similar ways before and it worked, but not sure why it's not working. FYI, this was a pivot table which I converted into pandas dataframe by using reset_index, but still showing error. The code I used,
# Convert pandas dataframe to spark data frame
spark_Forecast_report122 = spark.createDataFrame(df_top_six_dup1.astype(str))
# write the data into table
spark_Forecast_report122.write.mode("overwrite").saveAsTable("default.spark_Forecast_report122")
sdff_Forecast_report122 = spark.read.table("spark_Forecast_report122")
Forecast_Price_df122 = sdff_Forecast_report122.toPandas()
display(Forecast_Price_df122)
I am attaching the error a images.Image_1
The Dataframe is created using scala api for SPARK
val someDF = spark.createDataFrame(
spark.sparkContext.parallelize(someData),
StructType(someSchema)
)
I want to convert this to Pandas Dataframe
PySpark provides .toPandas() to convert a spark dataframe to pandas but there is no equivalent for scala(that I can find)
Please help me in this regard.
To convert a Spark DataFrame into a Pandas DataFrame, you can enable spark.sql.execution.arrow.enabled to true and then read/create a DataFrame using Spark and then convert it to Pandas DataFrame using Arrow
Enable spark.conf.set("spark.sql.execution.arrow.enabled", "true")
Create DataFrame using Spark like you did:
val someDF = spark.createDataFrame()
Convert the same to a pandas DataFrame
result_pdf = someDF.select("*").toPandas()
The above commands run using Arrow, because of the config spark.sql.execution.arrow.enabled set to true
Hope this helps!
In Spark DataFrame is just abstraction above data, most common sources of data are files from file system. When you convert dataframe in PySpark to Pandas format, PySpark just convert PySpark abstraction above data to another abstraction from another python framework. If you want made conversion in Scala between Spark and Pandas you can't do that because Pandas is Python library for work with data but spark is not and you will have some difficulties with Python and Scala integration. The best simple things you can do here:
Write dataframe to file system on scala Spark
Read data from file system using Pandas.
I'm trying to convert a Pandas DataFrame on each worker node (an RDD where each element is a Pandas DataFrame) into a Spark DataFrame across all worker nodes.
Example:
def read_file_and_process_with_pandas(filename):
data = pd.read(filename)
"""
some additional operations using pandas functionality
here the data is a pandas dataframe, and I am using some datetime
indexing which isn't available for spark dataframes
"""
return data
filelist = ['file1.csv','file2.csv','file3.csv']
rdd = sc.parallelize(filelist)
rdd = rdd.map(read_file_and_process_with_pandas)
The previous operations work, so I have an RDD of Pandas DataFrames. How can I convert this then into a Spark DataFrame after I'm done with the Pandas processing?
I tried doing rdd = rdd.map(spark.createDataFrame), but when I do something like rdd.take(5), i get the following error:
PicklingError: Could not serialize object: Py4JError: An error occurred while calling o103.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:272)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
Is there a way to convert Pandas DataFrames in each worker node into a distributed DataFrame?
See this question: https://stackoverflow.com/a/51231046/7964197
I've had to deal with the same problem, which seems quite common (reading many files using pandas, e.g. excel/pickle/any other non-spark format, and converting the resulting RDD into a spark dataframe)
The supplied code adds a new method on the SparkSession that uses pyarrow to convert the pd.DataFrame objects into arrow record batches which are then directly converted to a pyspark.DataFrame object
spark_df = spark.createFromPandasDataframesRDD(prdd) # prdd is an RDD of pd.DataFrame objects
For large amounts of data, this is orders of magnitude faster than converting to an RDD of Row() objects.
Pandas dataframes can not direct convert to rdd.
You can create a Spark DataFrame from Pandas
spark_df = context.createDataFrame(pandas_df)
Reference: Introducing DataFrames in Apache Spark for Large Scale Data Science