I have a python project with several modules, classes, and dependencies files (a requirements.txt file). I want to pack it into one file with all the dependencies and give the file path to AWS EMR serverless, which will run it.
The problem is that I don't understand how to pack a python project with all the dependencies, which file the EMR can consume, etc. All the examples I have found used one python file.
In simple words, what should I do if my python project is not a single file but is more complex?
Can anyone help with some details?
There's a few ways to do this with EMR Serverless. Regardless of which way you choose, you will need to provide a main entrypoint Python script to the EMR Serverless StartJobRun command.
Let's assume you've got a job structure like this where main.py is your entrypoint that creates a Spark session and runs your jobs and job1 and job2 are your local modules.
├── jobs
│ └── job1.py
│ └── job2.py
├── main.py
├── requirements.txt
Option 1. Use --py-files with your zipped local modules and --archives with a packaged virtual environment for your external dependencies
Zip up your job files
zip -r job_files.zip jobs
Create a virtual environment using venv-pack with your dependencies.
Note: This has to be done with a similar OS and Python version as EMR Serverless, so I prefer using a multi-stage Dockerfile with custom outputs.
FROM --platform=linux/amd64 amazonlinux:2 AS base
RUN yum install -y python3
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN python3 -m pip install --upgrade pip && \
python3 -m pip install venv-pack==0.2.0 && \
python3 -m pip install -r requirements.txt
RUN mkdir /output && venv-pack -o /output/pyspark_deps.tar.gz
FROM scratch AS export
COPY --from=base /output/pyspark_deps.tar.gz /
If you run DOCKER_BUILDKIT=1 docker build --output . ., you should now have a pyspark_deps.tar.gz file on your local system.
Upload main.py, job_files.zip, and pyspark_deps.tar.gz to a location on S3.
Run your EMR Serverless job with a command like this (replacing APPLICATION_ID, JOB_ROLE_ARN, and YOUR_BUCKET):
aws emr-serverless start-job-run \
--application-id $APPLICATION_ID \
--execution-role-arn $JOB_ROLE_ARN \
--job-driver '{
"sparkSubmit": {
"entryPoint": "s3://<YOUR_BUCKET>/main.py",
"sparkSubmitParameters": "--py-files s3://<YOUR_BUCKET>/job_files.zip --conf spark.archives=s3://<YOUR_BUCKET>/pyspark_deps.tar.gz#environment --conf spark.emr-serverless.driverEnv.PYSPARK_DRIVER_PYTHON=./environment/bin/python --conf spark.emr-serverless.driverEnv.PYSPARK_PYTHON=./environment/bin/python --conf spark.executorEnv.PYSPARK_PYTHON=./environment/bin/python"
}
}'
Option 2. Package your local modules as a Python library and use --archives with a packaged virtual environment
This is probably the most reliable way, but it will require you to use setuptools. You can use a simple pyproject.toml file along with your existing requirements.txt
[project]
name = "mysparkjobs"
version = "0.0.1"
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}
You then can use a multi-stage Dockerfile and custom build outputs to package your modules and dependencies into a virtual environment.
Note: This requires you to enable Docker Buildkit
FROM --platform=linux/amd64 amazonlinux:2 AS base
RUN yum install -y python3
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
WORKDIR /app
COPY . .
RUN python3 -m pip install --upgrade pip && \
python3 -m pip install venv-pack==0.2.0 && \
python3 -m pip install .
RUN mkdir /output && venv-pack -o /output/pyspark_deps.tar.gz
FROM scratch AS export
COPY --from=base /output/pyspark_deps.tar.gz /
Now you can run DOCKER_BUILDKIT=1 docker build --output . . and a pyspark_deps.tar.gz file will be generated with all your dependencies. Upload this file and your main.py script to S3.
Assuming you uploaded both files to s3://<YOUR_BUCKET>/code/pyspark/myjob/, run the EMR Serverless job like this (replacing the APPLICATION_ID, JOB_ROLE_ARN, and YOUR_BUCKET:
aws emr-serverless start-job-run \
--application-id <APPLICATION_ID> \
--execution-role-arn <JOB_ROLE_ARN> \
--job-driver '{
"sparkSubmit": {
"entryPoint": "s3://<YOUR_BUCKET>/code/pyspark/myjob/main.py",
"sparkSubmitParameters": "--conf spark.archives=s3://<YOUR_BUCKET>/code/pyspark/myjob/pyspark_deps.tar.gz#environment --conf spark.emr-serverless.driverEnv.PYSPARK_DRIVER_PYTHON=./environment/bin/python --conf spark.emr-serverless.driverEnv.PYSPARK_PYTHON=./environment/bin/python --conf spark.executorEnv.PYSPARK_PYTHON=./environment/bin/python"
}
}'
Note the additional sparkSubmitParameters that specify your dependencies and configure the driver and executor environment variables for the proper paths to python.
I try to use Syntaxnet with TensorFlow Serving.
I found this link here . I try to run the script to launch the server:
# you can create a shell script with content below!
git clone https://github.com/dsindex/syntaxnet.git work
cd work
git clone --recurse-submodules https://github.com/tensorflow/serving
# checkout proper version of serving
cd serving
git checkout 89e9dfbea055027bc31878ee8da66b54a701a746
git submodule update --init --recursive
# checkout proper version of tf_models
cd tf_models
git checkout a4b7bb9a5dd2c021edcd3d68d326255c734d0ef0
# you need to install gRPC properly
# https://tensorflow.github.io/serving/setup
# if you have a trouble, see https://github.com/dsindex/tensorflow#tensorflow-serving
# apply patch by dmansfield to serving/tf_models/syntaxnet
cd serving/tf_models
patch -p1 < ../../api/pr250-patch-a4b7bb9a.diff.txt
cd ../../
# configure serving/tensorflow
cd serving/tensorflow
./configure
cd ../../
# modify serving/tensorflow_serving/workspace.bzl for referencing syntaxnet
cp api/modified_workspace.bzl serving/tensorflow_serving/workspace.bzl
cat api/modified_workspace.bzl
# ...
# native.local_repository(
# name = "syntaxnet",
# path = workspace_dir + "/tf_models/syntaxnet",
# )
# ...
# append build instructions to serving/tensorflow_serving/example/BUILD
cat api/append_BUILD >> serving/tensorflow_serving/example/BUILD
# copy parsey_api.cc, parsey_api.proto to example directory to build
cp api/parsey_api* serving/tensorflow_serving/example/
# build parsey_api
cd serving
bazel --output_user_root=bazel_root build --nocheck_visibility -c opt -s //tensorflow_serving/example:parsey_api --genrule_strategy=standalone --spawn_strategy=standalone --verbose_failures
# make softlink for referencing 'syntaxnet/models/parsey_mcparseface/context.pbtxt'
ln -s ./tf_models/syntaxnet/syntaxnet syntaxnet
# run parsey_api with exported model
./bazel-bin/tensorflow_serving/example/parsey_api --port=9000 ../api/parsey_model
But for the line below (when building parsey_api):
bazel --output_user_root=bazel_root build --nocheck_visibility -c opt -s //tensorflow_serving/example:parsey_api --genrule_strategy=standalone --spawn_strategy=standalone --verbose_failures
I got an error:
Extracting Bazel installation...
........
ERROR: Failed to load Skylark extension '//tensorflow/tensorflow:workspace.bzl'.
It usually happens when the repository is not defined prior to being used.
Maybe repository '' was defined later in your WORKSPACE file?
ERROR: cycles detected during target parsing.
INFO: Elapsed time: 2.298s
./batch.sh: line 52: ./bazel-bin/tensorflow_serving/example/parsey_api: No such file or directory
The file workspace.bzl is in the directory $BUILD_DIRECTORY/tensorflow/tensorflow
I am using:
Ubuntu 14.04
Bazel 0.4.4
python gPRC installed
Thank you any suggestion is welcome.
I just created a docker container and tried to install SQL Relay inside it.
I've checked the prerequisites here and followed the installation documents here.
However, at the end of make install of sqlrelay, I saw an error like this:
update-rc.d: /etc/init.d/sqlrelay: file does not exist
update-rc.d: /etc/init.d/sqlrcachemanager: file does not exist
make[1]: *** [install] Error 1
make[1]: Leaving directory `/sqlrelay-0.66.0/init'
make: *** [install-init] Error 2
What might be wrong with my installation?
Here's the docker file I used to start my installation:
FROM ubuntu:trusty
RUN apt-get update && \
apt-get install libxml2-dev libpcre3 libpcre3-dev libmysqld-dev -y
RUN apt-get install mysql-server libmysqlclient-dev -y
# sql relay prerequisites
RUN apt-get install g++ make perl php5-dev python-dev ruby-dev \
tcl-dev openjdk-7-jdk erlang-dev nodejs-dev node-gyp mono-devel \
libmariadbclient-dev libpq-dev firebird-dev libfbclient2 libsqlite3-dev \
unixodbc-dev freetds-dev mdbtools-dev -y
COPY rudiments-0.56.0.tar.gz /
COPY sqlrelay-0.66.0.tar.gz /
EXPOSE 80
Here are the outputs of ./configure, make, and make install inside sqlrelay-0.66.0 folder:
configure_log
make_log
make_install_log
If you need more information of my installation process, just let me know. I can provide it.
I think you should use ADD instead of COPY in your lines such as
COPY rudiments-0.56.0.tar.gz /
Your COPY just copies the .tar.gz, but does not unpack them
as with ADD
If the <src> parameter of ADD is an archive in a recognised compression format, it will be unpacked
This is extracted from
What is the difference between the `COPY` and `ADD` commands in a Dockerfile?
I have recently hit the same issue. The issue I found was that the init Makefile was incorrectly detecting the use of systemctl on Ubuntu Trusty and putting the scripts there. Later on the script would try to find the scripts in init.d and fail.
The solution is to edit the Makefile: sqlrelay-X.X.X/init/Makefile
Replace:
install:
if ( test -d "/lib/systemd/system" ); \
With:
install:
if ( test -d "/lib/systemd/system_x" ); \
Make a similar change to the uninstall option later in the script and it will now correctly install on Ubuntu.
I wrote myself a litte script to install opencv under ubuntu14.04. Can I remove the directory 3party after the make install sorted the lib into system directories or are there dependencies? (Remove not only the MYBUILD but the complete 3party)
echo "\nInstall OpenCV?...<any key>\n"
read inp1; # $inp1
mkdir 3party;
cd 3party;
git clone https://github.com/Itseez/opencv.git
cd opencv;
mkdir MYBUILD;
cd MYBUILD;
#sudo mkdir -p /usr/local/lib/opencv;
cmake -L -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local .. ;
echo"check if path is ok?...<any key> or abort";
read inp1; # $inp1
make;
#sudo mkdir -p /usr/local/lib/opencv;
make install;
cd ../../..;
chmod -R 777 3party;
echo "\nDone.\nPlease exit...<any key>";
EDIT: I did tag it cmake because the configuration step is performed with this build tool. Also the tutorial on the OpenCV website stated it. Please correct me if wrong.
Building OpenCV from Source Using CMake, Using the Command Line
Normally, after installation of any package its source and binary directories can be safetly removed. OpenCV follows this convention too.
I am trying to install Apache HTTP server locally in my box as a regular user (non-root).
I have downloaded Apache 2.4.1 version of Apache HTTP server [http://httpd.apache.org/download.cgi]. However when I am trying to build and install locally in my box I am getting below error:
httpd/httpd-2.4.1 1059> ./configure
checking for chosen layout... Apache
checking for working mkdir -p... yes
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
Configuring Apache Portable Runtime library ...
checking for APR... no
configure: error: APR not found. Please read the documentation.
I am not sure what dependency it is looking for - I mean the download package does not contains it? What I need to do to build / deploy Apache HTTP server?
When it tells you Please read documentation it means that you should go read Apache documentation ( http://httpd.apache.org/docs/2.4/install.html ) which tells you to
download the latest versions of both APR and APR-Util from Apache APR,
unpack them into ./srclib/apr and ./srclib/apr-util (be sure the
domain names do not have version numbers; for example, the APR
distribution must be under ./srclib/apr/)
then do
./configure --with-included-apr
For Ubuntu 11.10 this option seems to be working well:
# APR
wget http://mirrors.axint.net/apache//apr/apr-1.4.6.tar.gz
tar -xvzf apr-1.4.6.tar.gz
cd apr-1.4.6/
./configure
make
make install
cd ..
# APR Utils
wget http://mirrors.axint.net/apache//apr/apr-util-1.4.1.tar.gz
tar -xvzf apr-util-1.4.1.tar.gz
cd apr-util-1.4.1
./configure --with-apr=/usr/local/apr
make
make install
cd ..
# Apache
wget http://apache.petsads.us//httpd/httpd-2.4.1.tar.gz
tar -xvzf httpd-2.4.1.tar.gz
cd httpd-2.4.1
./configure --enable-file-cache --enable-cache --enable-disk-cache --enable-mem-cache --enable-deflate --enable-expires --enable-headers --enable-usertrack --enable-ssl --enable-cgi --enable-vhost-alias --enable-rewrite --enable-so --with-apr=/usr/local/apr/
make
make install
cd ..
You can find more about it below
Source: VaporCreations.com
If you have Debian/Ubuntu you can just:
apt-get install libapr1-dev libaprutil1-dev
Then ./configure
Done
Here are the steps of how I installed apache-httpd on a non-root users:
Download and extract apache-httpd-2.4.2 (but before ./configuring,
making and installing it, follow the steps below:)
Download and extract APR & APR-UTIL into "./srclib/apr" &
"./srclib/apr-util" folders. This requires ./configure
--with-apr=./apache/httpd-2.4.2/srclib/apr (and) --with-included-apr (options).
Download, extract, ./configure (with) --prefix=localURL, make and
make install PCRE into "./pcre" folder. This requires ./configure
--with-pcre=/home/username/apache/pcre (option).
Configure apache-httpd by entring following command (I like to
enable certain options as written in the command below):
./configure --enable-file-cache --enable-cache --enable-disk-cache
--enable-mem-cache --enable-deflate --enable-expires --enable-headers --enable-usertrack --enable-cgi --enable-vhost-alias --enable-rewrite --enable-so --with-apr=/home/username/apache/httpd-2.4.2/srclib/apr --prefix=/home/username/apache/httpd-2.4.2/ --with-included-apr --with-pcre=/home/username/apache/pcre
Note: When configuring apache-httpd, use option "--enable-ssl" ONLY if OpenSSL is installed otherwise DON'T enable it.
Now on the command-line, enter 'make' and 'make install' command.
Open and configure the 'httpd.conf' file e.g.:
"vi /home/eddie_kumar/apache/httpd-2.4.2/conf/httpd.conf"
IMPORTANT: don't forget to change the default port from 80 to something else e.g. 8080, this is especially important for non-root
user. (How to? open httpd.conf -> search "Listen 80" -> change it
to "Listen 8080".
And that's it, now open your browser enter "localhost:8080", it should display "It works!".
If you are using fedora, you can use yum to install APR, APR-Util and PCRE. You'll also need to download apr-devel, apr-util-devel and pcre-devel.
That being said, you can just run the following command on your terminal and no more "configure: error: APR not found.. ..APR-Util and PCRE" errors.
yum -y install arp apr-devel apr-util apr-util-devel pcre pcre-devel
I'm using fedora 17 and planning on using a shell script to setup apache 2.4.3. So yum works pretty slick instead of manually downloading apr, apr-util, and pcre.
1, You need APR (apache portable runtime), which is core component of apache web server
2, If you wnat to do make install, you may need root account
3, Even if not, apache can not start to listen on unprivileged port (lower then 1024) without root account
4, gain root or ask someone with root to install apache from official repo(I don;t know which distro you run) like using yum, apt-get, etc...
Apr or pcre related errors require the source to be downloaded and Apache HTTPD "configure" process needs to be made aware of these source locations on your file system. For instance: if you downloaded the source for APR at ./srclib (relative to apache httpd) then you would use
--with-included-apr
as the configure option.
On the other hand if you want to not build but install APR / APR-UTIL, then you need the following on CentOS / RedHat:
yum install apr-util-devel apr-devel
However it might so happen that the APR version provided by yum does not match what is expected by this version of Apache httpd. In that case you could download APR and APR-UTIL and use the --with-included-apr option.
You could also build PCRE utilizing the same "configure, make, make install" process and then continue where you left off building Apache httpd.
Or you could install pcre:
yum install pcre-devel
If while building PCRE: you see "compile: unrecognized option" then perhaps you would need other dependencies as well: Please see the details at:
http://khanna111.com/wordPressBlog/2012/09/11/94087-2/
It also covers "mod_deflate" and "zlib" as well.
Basic steps
tar -xvf httpd-2.4.1-customized.tar -C ../
#Balancer folder will be created
tar -xvzf openssl-1.0.1.tar.gz -C /balancer/
cd ->/balancer/openssl-1.0.1
./config --prefix=/usr/local/ssl/ shared zlib-dynamic enable-camellia
make depend
make
make install
tar -xvzf pcre-8.30.tar.gz -C ../balancer/
/balancer/pcre-8.30
./configure --prefix=/usr/local/pcre/
make
make install
remove pcre and openssl
Installing and compiling the Apache server on Ubuntu machine
step 1:Install the Java JDK
sudo apt install openjdk-11-jdk
sudo gedit /etc/environment
JAVA_HOME="/usr/lib/jvm/openjdk-11"
source /etc/environment
echo $JAVA_HOME
verify the java version
javac --version
**** install Other required packages:**
sudo apt-get install apache2-dev -y
sudo apt-get install libpcre3 libpcre3-dev
Step 2 :To Install Apache HTTP Server
#1 For ubuntu ,install development tools including the C compiler:
sudo apt-get install build-essential checkinstall
#2 Download and Extract the required files
assuming that all the files have been downloaded into the ~/Downloads directory
Download the Apache HTTP Server httpd-2.4.41.tar.gz from Apache download page (https://httpd.apache.org/download.cgi)
and following required libraries for compiling the apache HTTP server:
apr-1.7.0.tar.gz(http://apr.apache.org/download.cgi)
apr-util-1.6.1.tar.gz(http://apr.apache.org/download.cgi)
pcre2-10.34.tar.gz (ftp://ftp.pcre.org/pub/pcre/) http://pcre.org/
****Read Requirements Section(http://httpd.apache.org/docs/2.4/install.html)***
#3 Extract the tar files
tar -xvf httpd-2.4.41.tar.gz
tar -xvf apr-1.7.0.tar.gz
tar -xvf apr-util-1.6.1.tar.gz
tar -xvf pcre2-10.34.tar.gz
after extraction you should see following list of directories in ~/Downloads
httpd-2.4.41
apr-1.7.0
apr-util-1.6.1
pcre2-10.34
#4 Create a directory for the apache HTTP Server
*make sure to give all rights to this directory so that while compiling files can be read/written
sudo mkdir /home{your username here}/apache
To give all permissions to a apache directory :
sudo chmod -R 777 /home{your username here}/apache
#5 copy the arp and arp-util directory into the ~/Downloads/httpd-2.4.41/srclib/
cd ~/Downloads
mv apr-util-1.6.1 ~/Downloads/httpd-2.4.41/srclib/apr-util
mv apr-1.7.0 ~/Downloads/httpd-2.4.41/srclib/apr
#6 Configure the sources for compilation.
The --prefix option can be used to install the Web server in a location where you can write files.
cd ~/Downloads/httpd-2.4.41/
./configure --prefix=/home/{username here}/apache --with-pcre=~/Downloads/pcre2-10.34
NOTE:
If you have some problems while running above command,you can also try
insallting pcre in /usr/local/pcre, using
./configure --prefix=/usr/local/pcre
make
sudo make install
commands and then by running folllowing commands:
cd ~/Downloads/httpd-2.4.41/
./configure --prefix=/home/{username here}/apache --with-pcre=/usr/local/pcre
make
sudo make install
If you still face some problems ,make sure that ~/apache and its nested directory have read/write permissions. if not run **sudo chmod -R 777 /home{your username here}/apache** command again.
#7 Compile Apache HTTP Server.
cd ~/Downloads/httpd-2.2.25
sudo make
#8 Install Apache HTTP Server.
cd ~/Downloads/httpd-2.2.25
sudo make install
Optional
#9 To Prepare Your Hosts File
sudo gedit /etc/hosts
27.0.0.1 localhost www.example.com
sudo gedit /home/{your username here}/apache/conf/httpd.conf
and copy:
Listen 8000
ServerName www.example.com:8000
#10 Test the installation to ensure Apache HTTP Server is working.
/home/{your username here}//apache/bin/apachectl -k start