Using Syntaxnet with TensorFlow Serving - tensorflow

I try to use Syntaxnet with TensorFlow Serving.
I found this link here . I try to run the script to launch the server:
# you can create a shell script with content below!
git clone https://github.com/dsindex/syntaxnet.git work
cd work
git clone --recurse-submodules https://github.com/tensorflow/serving
# checkout proper version of serving
cd serving
git checkout 89e9dfbea055027bc31878ee8da66b54a701a746
git submodule update --init --recursive
# checkout proper version of tf_models
cd tf_models
git checkout a4b7bb9a5dd2c021edcd3d68d326255c734d0ef0
# you need to install gRPC properly
# https://tensorflow.github.io/serving/setup
# if you have a trouble, see https://github.com/dsindex/tensorflow#tensorflow-serving
# apply patch by dmansfield to serving/tf_models/syntaxnet
cd serving/tf_models
patch -p1 < ../../api/pr250-patch-a4b7bb9a.diff.txt
cd ../../
# configure serving/tensorflow
cd serving/tensorflow
./configure
cd ../../
# modify serving/tensorflow_serving/workspace.bzl for referencing syntaxnet
cp api/modified_workspace.bzl serving/tensorflow_serving/workspace.bzl
cat api/modified_workspace.bzl
# ...
# native.local_repository(
# name = "syntaxnet",
# path = workspace_dir + "/tf_models/syntaxnet",
# )
# ...
# append build instructions to serving/tensorflow_serving/example/BUILD
cat api/append_BUILD >> serving/tensorflow_serving/example/BUILD
# copy parsey_api.cc, parsey_api.proto to example directory to build
cp api/parsey_api* serving/tensorflow_serving/example/
# build parsey_api
cd serving
bazel --output_user_root=bazel_root build --nocheck_visibility -c opt -s //tensorflow_serving/example:parsey_api --genrule_strategy=standalone --spawn_strategy=standalone --verbose_failures
# make softlink for referencing 'syntaxnet/models/parsey_mcparseface/context.pbtxt'
ln -s ./tf_models/syntaxnet/syntaxnet syntaxnet
# run parsey_api with exported model
./bazel-bin/tensorflow_serving/example/parsey_api --port=9000 ../api/parsey_model
But for the line below (when building parsey_api):
bazel --output_user_root=bazel_root build --nocheck_visibility -c opt -s //tensorflow_serving/example:parsey_api --genrule_strategy=standalone --spawn_strategy=standalone --verbose_failures
I got an error:
Extracting Bazel installation...
........
ERROR: Failed to load Skylark extension '//tensorflow/tensorflow:workspace.bzl'.
It usually happens when the repository is not defined prior to being used.
Maybe repository '' was defined later in your WORKSPACE file?
ERROR: cycles detected during target parsing.
INFO: Elapsed time: 2.298s
./batch.sh: line 52: ./bazel-bin/tensorflow_serving/example/parsey_api: No such file or directory
The file workspace.bzl is in the directory $BUILD_DIRECTORY/tensorflow/tensorflow
I am using:
Ubuntu 14.04
Bazel 0.4.4
python gPRC installed
Thank you any suggestion is welcome.

Related

gitlab-ci job not running script

I am new to gitlab-ci and trying a minimal Python application based on a gitlab template.
My .gitlab-ci.yml file is below:
# This file is a template, and might need editing before it works on your project.
# Official language image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python/tags/
#image: python:latest
# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache"
stages:
- test
- run
# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/reference/pip_install/#caching
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
#cache:
# paths:
# - .cache/pip
# - venv/
before_script:
- python -V # Print out python version for debugging
#- pip install virtualenv
#- virtualenv venv
- python -m venv venv
- venv/scripts/activate
job1:
stage: test
script:
- python setup.py test
#- pip install tox flake8 # you can also use tox
#- tox -e py36,flake8
job2:
stage: run
script:
- pip install wheel
- python setup.py bdist_wheel
artifacts:
paths:
- dist/*.whl
#pages:
# script:
# - pip install sphinx sphinx-rtd-theme
# - cd doc ; make html
# - mv build/html/ ../public/
# artifacts:
# paths:
# - public
# only:
# - master
The jobs are seen within the gitlab web UI and they appear to run on my (windows based shell executor) runner.
When I look at the output for the jobs, it appears as if the actual script commands for each job aren't running at all.
Here's the output from job1:
Running with gitlab-runner 11.2.0 (35e8515d)
on GKUHN-L04 b0162458
Using Shell executor...
Running on GKUHN-L04...
Fetching changes...
Removing venv/
HEAD is now at 2484105 And agai..
From https://gitlab.analog.com/GKuhn/test_gitlab_ci
- [deleted] (none) -> origin/test_ci
fdd4216..cd618ba master -> origin/master
Checking out cd618ba9 as master...
Skipping Git submodules setup
$ python -V
Python 3.7.0
$ python -m venv venv
$ venv/scripts/activate
Job succeeded
And job2:
Running with gitlab-runner 11.2.0 (35e8515d)
on GKUHN-L04 b0162458
Using Shell executor...
Running on GKUHN-L04...
Fetching changes...
Removing venv/
HEAD is now at cd618ba updated .gitlab-ci.yml file
Checking out cd618ba9 as master...
Skipping Git submodules setup
$ python -V
Python 3.7.0
$ python -m venv venv
$ venv/scripts/activate
Uploading artifacts...
WARNING: dist/*.whl: no matching files
ERROR: No files to upload
Job succeeded
What am I doing wrong??
Turns out this is bug on Windows with the shell executor:
https://gitlab.com/gitlab-org/gitlab-runner/issues/2730
And a duplicate of this question: Gitlab CI does not execute npm scripts

Building Apache Impala fails

I was trying to build Apache Impala from source(newest version on github).
I followed following instructions to build Impala:
(1) clone Impala
> git clone https://git-wip-us.apache.org/repos/asf/incubator-impala.git
> cd Impala
(2) configure environmental variables
> export JAVA_HOME=/usr/lib/jvm/java-7-oracle-amd64
> export IMPALA_HOME=<path to Impala>
> export BOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu
> export LC_ALL="en_US.UTF-8"
(3)build
${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
(4) errors are shown below:
Heap is needed to find the cause. Looks like the compiler does not support the GLIBCXX_3.4.21. But the GCC is automatically downloaded by the building script.
Appreciate your help!!!
Starting from this commit https://github.com/apache/impala/commit/d5cefe07c931a0d3bf02bca97bbba05400d91a48 , Impala has been shipped with a development bootstrap script.
I tried the master branch in a fresh ubuntu 16.04 docker image and it works fine. Here is what I did.
checkout the latest impala code base and do
docker run --rm -it --privileged -v /home/amos/git/impala/:/root/Impala ubuntu:16.04
inside docker, do
apt-get update
apt-get install sudo
cd /root/Impala
comment this out in bin/bootstrap_system.sh if you don't need test data
# if ! [[ -d ~/Impala-lzo ]]
# then
# git clone https://github.com/cloudera/impala-lzo.git ~/Impala-lzo
# fi
# if ! [[ -d ~/hadoop-lzo ]]
# then
# git clone https://github.com/cloudera/hadoop-lzo.git ~/hadoop-lzo
# fi
# cd ~/hadoop-lzo/
# time -p ant package
also add this line before ssh localhost whoami
echo "source ${IMPALA_HOME}/bin/impala-config-local.sh" >> ~/.bashrc
change the build command to whatever you like in bin/bootstrap_development.sh
${IMPALA_HOME}/buildall.sh -noclean -skiptests -build_shared_libs -format
then run bin/bootstrap_development.sh
You'll be prompted for some input. Just fill in default value and it'll work.

installing TensorFlow on the IBM power8

I have access to a large IBM Power8 machine, and would like to install TensorFlow on it. Naturally, I tried the quick pip install, but it failed:
sudo pip install https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp27-none-linux_x86_64.whl
tensorflow-0.6.0-cp27-none-linux_x86_64.whl is not a supported wheel on this platform.
Storing debug log for failure in /home/pv/.pip/pip.log
Unfortunately, pip.log cotains little useful info.
/usr/bin/pip run on Sat Feb 6 17:29:34 2016
tensorflow-0.6.0-cp27-none-linux_x86_64.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
InstallRequirement.from_line(name, None))
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 168, in from_line
raise UnsupportedWheel("%s is not a supported wheel on this platform." % wheel.filename)
UnsupportedWheel: tensorflow-0.6.0-cp27-none-linux_x86_64.whl is not a supported wheel on this platform.
Next thing I tried was to build TensorFlow from source. To no avail, all my attempts ended with some cannot execute binary file: Exec format error message, e.g.:
/usr/local/bin/bazel: line 86: /usr/local/lib/bazel/bin/bazel-real: cannot execute binary file: Exec format error
So then I tried to compile Bazel from source, which also resulted in a similar hard error.
me#machine:~/bazel-0.1.5$ ./compile.sh
INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO: ./compile.sh compile /path/to/bazel
🍃 Building Bazel from scratch.
Compiling Java stubs for protocol buffers...
third_party/protobuf/protoc-linux-x86_32.exe -Isrc/main/protobuf/ --java_out=/tmp/bazel.T9C83cNa/src src/main/protobuf/android_studio_ide_info.proto
scripts/bootstrap/buildenv.sh: line 63: third_party/protobuf/protoc-linux-x86_32.exe: cannot execute binary file: Exec format error
pv#sardonis:~/bazel-0.1.5$ ^C
I however found this link http://www.cnblogs.com/rodenpark/p/5007744.html that explains how to build the Protobuf compiler from source on the Power8 machine. This worked and after the modifications described in his other topic http://www.cnblogs.com/rodenpark/p/5007846.html I managed to at least get the compilation process started. But now it crashes with a ton of errors which each seem less severe on their own but the vast amount of them makes it look really hopeless, I posted them on http://pastebin.com/KjkseaGx for reference.
So... I'm running out of inspiration. What can I do to make TensorFlow work on the Power8 machine?
Install bazel 0.2.0-ppc
tf#ubuntu16:~$ git clone https://github.com/ibmsoe/bazel
tf#ubuntu16:~/bazel$ git checkout v0.2.0-ppc
tf#ubuntu16:~/bazel$ ./compile.sh
Install tensorflow
tf#ubuntu16:~$ git clone --recurse-submodules https://github.com/tensorflow/tensorflow
tf#ubuntu16:~/tensorflow$ git checkout v0.10.0rc0
tf#ubuntu16:~/tensorflow$ git commit -m"v0.10.0rc0"
tf#ubuntu16:~/tensorflow$ git cherry-pick ce70f6cf842a46296119337247c24d307e279fa0
tf#ubuntu16:~/tensorflow$ git cherry-pick f1acb3bd828a73b15670fc8019f06a5cd51bd564
tf#ubuntu16:~/tensorflow$ git cherry-pick 9b6215a691a2eebaadb8253bd0cf706f2309a0b8
tf#ubuntu16:~/tensorflow$ ./configure
tf#ubuntu16:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
Here you'll encounter an error, something like this
ERROR: /home/tf/.cache/bazel/_bazel_tf/b2f766da603b0bed56d4c1d0b178456a/external/farmhash_archive/BUILD:5:1: Executing genrule #farmhash_archive//:configure failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
/home/tf/.cache/bazel/_bazel_tf/b2f766da603b0bed56d4c1d0b178456a/tensorflow/external/farmhash_archive/farmhash-34c13ddfab0e35422f4c3979f360635a8c050260 /home/tf/.cache/bazel/_bazel_tf/b2f766da603b0bed56d4c1d0b178456a/tensorflow
/tmp/tmp.XdCPQefJyZ /home/tf/.cache/bazel/_bazel_tf/b2f766da603b0bed56d4c1d0b178456a/tensorflow/external/farmhash_archive/farmhash-34c13ddfab0e35422f4c3979f360635a8c050260 /home/tf/.cache/bazel/_bazel_tf/b2f766da603b0bed56d4c1d0b178456a/tensorflow
You'll have to edit config.guess as below to insert a stanza for ppc64le
tf#ubuntu16:~/.cache/bazel/_bazel_tf/b2f766da603b0bed56d4c1d0b178456a/external/farmhash_archive/farmhash-34c13ddfab0e35422f4c3979f360635a8c050260$ vi config.guess
*:BSD/OS:*:*)
echo ${UNAME_MACHINE}-unknown-bsdi${UNAME_RELEASE}
exit ;;
+ ppc64le:Linux:*:*)
+ echo powerpc64le-unknown-linux-gnu
+ exit ;;
*:FreeBSD:*:*)
case ${UNAME_MACHINE} in
tf#ubuntu16:~/tensorflow$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
tf#ubuntu16:~/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
tf#ubuntu16:~/tensorflow$ sudo pip install /tmp/tensorflow_pkg/tensorflow*.whl
tf#ubuntu16:~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfi
tf#ubuntu16:~/tensorflow$ mkdir _python_build
tf#ubuntu16:~/tensorflow$ cd _python_build
tf#ubuntu16:~/tensorflow/_python_build$ ln -s ~/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/tensorflow/* .
tf#ubuntu16:~/tensorflow/_python_build$ ln -s ~/tensorflow/tools/* .
tf#ubuntu16:~/tensorflow/_python_build$ python __init__.py develop
Using miniconda:
Installing miniconda:
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux--ppc64le.sh -O miniconda.sh
bash miniconda.sh
Accept the condition and allow conda to be added to PATH
rm miniconda.sh
echo export IBM_POWERAI_LICENSE_ACCEPT=yes >> ~/.bashrc
source ~/.bashrc
This sould add (base) on terminal. Add the correct channel as first priority
conda config --add default_channels https://repo.anaconda.com/pkgs/main
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
Create environment (it is good practice not to install packages on base)
conda create -n ai python=3.7
conda activate ai
conda install --strict-channel-priority tensorflow-gpu
For more information on miniconda on IBM Power 8 and Anaconda: IBM Source & Anaconda Source

Can I remove directory after $git clone and $make install

I wrote myself a litte script to install opencv under ubuntu14.04. Can I remove the directory 3party after the make install sorted the lib into system directories or are there dependencies? (Remove not only the MYBUILD but the complete 3party)
echo "\nInstall OpenCV?...<any key>\n"
read inp1; # $inp1
mkdir 3party;
cd 3party;
git clone https://github.com/Itseez/opencv.git
cd opencv;
mkdir MYBUILD;
cd MYBUILD;
#sudo mkdir -p /usr/local/lib/opencv;
cmake -L -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local .. ;
echo"check if path is ok?...<any key> or abort";
read inp1; # $inp1
make;
#sudo mkdir -p /usr/local/lib/opencv;
make install;
cd ../../..;
chmod -R 777 3party;
echo "\nDone.\nPlease exit...<any key>";
EDIT: I did tag it cmake because the configuration step is performed with this build tool. Also the tutorial on the OpenCV website stated it. Please correct me if wrong.
Building OpenCV from Source Using CMake, Using the Command Line
Normally, after installation of any package its source and binary directories can be safetly removed. OpenCV follows this convention too.

Compile mod_cluster with httpd 2.4.17 - no rules.mk created

I need to compile for the modules from mod_cluster using https 2.4.17 and I am having an issue compiling the modules. The process fails at the make step.
I have successfully build httpd 2.4.17 into and rpm and installed it without issue.
I am pulling the mod_cluster from source at: https://github.com/modcluster/mod_cluster
I am following procedure to build mod_cluster:
cd /mod_cluster//native/advertise # Advertise the first of four modules
./buildconf
./configure --with-apxs=/usr/bin/apxs
checking for Apache httpd installation... APXS is /usr/bin/apxs
apxs_support is true
Use of uninitialized value in concatenation (.) or string at /usr/bin/apxs line 222.
configure: creating ./config.status
config.status: creating Makefile
make
Makefile:10: //build/rules.mk: No such file or directory
make: *** No rule to make target `//build/rules.mk'. Stop.
I believe the issue with the top_builddir directive in the make file.
Note: That there is not /build/rules.mk being written to /
# Makefile.in for mod_proxy_cluster
# copy the source in the httpd Apache source tree
APACHE_BASE = /usr
top_builddir = /
# For .deps.
builddir = /srv/apache/mod_cluster-master/native/advertise
# For the apache includes
top_srcdir = /usr
include $(top_builddir)/build/rules.mk
SH_COMPILE = $(LIBTOOL) --mode=compile $(BASE_CC) -I../include -prefer-pic -c $< && touch $#
all: mod_advertise.so
mod_advertise.so: mod_advertise.la
$(top_builddir)/build/instdso.sh SH_LIBTOOL='$(LIBTOOL)' mod_advertise.la `pwd`
mod_advertise.la: mod_advertise.slo
$(SH_LINK) -rpath $(libexecdir) -module -avoid-version mod_advertise.lo
clean:
rm -f *.o *.lo *.slo *.so
rm -rf .libs
Thank you
mod_cluster master compiles just fine with httpd 2.4.17. You might take a look at my Dockerfile that shows how it is done. I've just triggered a new DockerHub build, it will be available on DockerHub eventually.
In order to be absolutely sure, I repeated the process on Fedora22 x86_64 a minute ago manually:
wget http://archive.apache.org/dist/httpd/httpd-2.4.17.tar.gz
wget http://archive.apache.org/dist/httpd/httpd-2.4.17-deps.tar.gz
tar xvf ..., cd ...
./configure --prefix=/opt/httpd-2.4.17-build --with-mpm=worker --enable-mods-shared=most --enable-maintainer-mode --with-expat=builtin --enable-ssl --enable-proxy --enable-proxy-http --enable-proxy-ajp --with-threads
git clone https://github.com/modcluster/mod_cluster.git
cd mod_cluster/native
modules="advertise mod_cluster_slotmem mod_manager mod_proxy_cluster";for module in $modules;do cd $module;./buildconf;./configure --with-apxs=/opt/httpd-2.4.17-build/bin/apxs;make clean;make;cd ..;done;
So, apparently, the problem lies with your httpd build. Could you share your src rpm?