Can't build spark py-files with pandas included - pandas

Question

I am attempting to package up my dependencies for a spark program I am creating. I have a requirements.txt file as below
pandas
I then run
pip3 install -t dependencies -r requirements.txt
cd dependencies
zip -r ../dependencies.zip .
pyspark --py-files dependencies.zip
And run the line -
import pandas
And I get the error -
Traceback (most recent call last):
File "/mnt/tmp/spark-REDACTED/userFiles-REDACTED/dependencies.zip/pandas/__init__.py", line 31, in <module>
File "/mnt/tmp/spark-REDACTED/userFiles-REDACTED/dependencies.zip/pandas/_libs/__init__.py", line 3, in <module>
File "/mnt/tmp/spark-REDACTED/userFiles-REDACTED/dependencies.zip/pandas/_libs/tslibs/__init__.py", line 3, in <module>
ModuleNotFoundError: No module named 'pandas._libs.tslibs.conversion'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/tmp/spark-REDACTED/userFiles-REDACTED/dependencies.zip/pandas/__init__.py", line 36, in <module>
ImportError: C extension: No module named 'pandas._libs.tslibs.conversion' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.
Any ideas on how to fix this?

Answer 1

In order to ship dependency on the worker, there are two ways one is exactly what you did, zip the file, or simple py file then use --py-file. The problem you encountered is because of missing C dependency on the worker side. Pkg like NumPy/pandas all have c dependency.
In order to solve this, create the virtualenv, and zip the virtualenv including
the python executable
PYSPARK_DRIVER_PYTHON = <path to current working python>
PYSPARK_PYTHON = './venv/<path to python executable>'
pyspark --archives = <path to zip file>#venv
or follow this link