Bazel use tensorflow ( or only use tbb) as a external dependency has a "Make" Variables not defined error - tensorflow

I have a c++ project that use bazel as build tools and use tensorflow as a external dependency, bazel version is 0.24.1 and tensorflow version is 1.14.0, it works fine (In my project I use tbb in the tensorflow as a deps in bazel's rule).
But when I update bazel to 3.1.0 and update tensorflow to 2.3.0, I have the error:
ERROR: external/tbb/BUILD.bazel:11:1: in cmd attribute of genrule rule #tbb//:build_tbb: $(AR) not defined
My WOKSPACE file:
http_archive(
name = "org_tensorflow",
strip_prefix = "tensorflow-2.3.0",
sha256 = "2595a5c401521f20a2734c4e5d54120996f8391f00bb62a57267d930bce95350",
urls = ["https://github.com/tensorflow/tensorflow/archive/tensorflow-2.3.0.tar.gz"]
)
My BUILD file:
cc_library(
name = "tf_utils",
srcs = ["tf_utils.cpp"],
hdrs = ["tf_utils.h"],
deps = ["#org_tensorflow//tensorflow/cc/saved_model:loader", "utils"],
visibility = ["//visibility:public"]
)
cc_library(
name = "utils",
hdrs = ["utils.h"],
srcs = ["utils.cpp"],
deps = ["#boost//:filesystem",
"#eigen_archive//:eigen",
"#tbb//:tbb",], # I use the tbb in the tensorflow
visibility = ["//visibility:public"]
)
Firts, I test tbb in the tensorflow project itself, it has the same error.
The way of my test is, write the follow c++ rule in the tensorflow/BUILD:
cc_library(
name = "test-tbb",
srcs = ["test.cpp"],
deps = ["#ngraph_tf//:ngraph_tf"] # ngraph_tf's deps has tbb
)
and then excute bazel build //tensorflow:test-tbb, it has the same error:
#tbb//:build_tbb: $(AR) not defined
second, I think the problem may be in tbb. I test tbb in two ways:
I download tbb sources, and in the root dir of the tbb, make a WORKSPACE(empty) and a BUILD (only rename tbb.BUILD to BUILD), then excute bazel build //:build_tbb, when bazel version is 0.24.1, all is right, but has the above error when change bazel version to 3.1.0.
I only use tbb as a external dependency in a miniproject, which WORKSPACE has a http_archive named tbb, and it's build_file point to a tbb.BUILD. I write a cc_library in BUILD flie and it's deps = ["#tbb:build_tbb"], the test result is the same as 1), when bazel version is 0.24.1 is ok, 3.1.0 is broken.
tbb.BUILD I used: https://github.com/tensorflow/tensorflow/blob/r2.3/third_party/ngraph/tbb.BUILD
Third, I follow the document of bazel (https://docs.bazel.build/versions/master/be/make-variables.html#custom_variables), sets toolchains = ["#bazel_tools//tools/cpp:current_cc_toolchain"] in the rule "build_tbb" in tbb.BUILD, than the error message is "in cmd attribute of genrule rule //:build_tbb: $(CC_FLAGS) not defined".
Fourth, I follow the solution of the issue:https://github.com/rnburn/satyr/issues/2, update the tbb.BUILD file, than retry and the error message is "in cmd attribute of genrule rule #tbb//:build_tbb: $(C_COMPILER) not defined".
I just want use use tensorflow as an external dependency and use the bazel target "tbb" statement in tensorflow (Since it is in tensorflow, I don’t want to recompile multiple times)
Does anyone know how to correctly use tensorflow and tbb as an external dependency that use bazel 3.0.0+? Thanks!

Related

How to use bazel to build tensorflow as external dependencies to support gpu inference?

I want to use tensorflow's c++ api to inference in my project, when I build it for cpu, it's ok. But when I build it for support gpu (bazel build --config=cuda ...), it has some error.
My WORKSPACE:
http_archive(
name = "org_tensorflow",
strip_prefix = "tensorflow-1.15.0",
sha256 = "a5d49c00a175a61da7431a9b289747d62339be9cf37600330ad63b611f7f5dc9",
urls = ["https://github.com/tensorflow/tensorflow/archive/v1.15.0.tar.gz"]
)
load("#org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")
tf_workspace(tf_repo_name = "org_tensorflow")
# Uncomment the following two lines has the same error.
# load("#org_tensorflow//third_party/gpus:cuda_configure.bzl", "cuda_configure")
# cuda_configure(name="local_config_cuda")
My .bazelrc:
...
# config for using cuda
build:cuda --define=using_cuda=true --define=using_cuda_nvcc=true
build:cuda --action_env TF_NEED_CUDA=1
build:cuda --crosstool_top=#local_config_cuda//crosstool:toolchain
...
My BUILD:
cc_library(
name = "tf_utils",
srcs = ["tf_utils.cpp"],
hdrs = ["tf_utils.h"],
deps = ["#org_tensorflow//tensorflow/cc/saved_model:loader"
"#org_tensorflow//tensorflow/core:tensorflow"]
)
cc_binarary(
name = "test",
srcs = ["test.cpp"],
deps = ["tf_utils"]
)
When I run bazel build --config=cuda //:test
The error message is:
ERROR: external/com_google_protobuf/BUILD:405:1: C++ compilation of rule '#com_google_protobuf//:protoc' failed (Exit 1) crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/host/bin/external/com_google_protobuf/_objs/protoc/main.d ... (remaining 46 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
src/main/tools/linux-sandbox-pid1.cc:413: "execvp(external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc, 0x6dfe10)": No such file or directory
Target //:test failed to build
Use --verbose_failures to see the command lines of failed build steps.
The error message shows no such file or directory, but it does exist.
# ls -la external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc
-rwxrwxr-x 1 root root 9129 Dec 1 14:14 external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc*
The tensorflow version is 1.15, bazel version is 3.1.0.
Does anyone know how to configure bazel correctly so that tensorflow can support gpu as an external dependency? Thanks very much.

bazel tensorflow target name misunderstanding

I try to understand bazel dependency tree in tensorflow 2.0 project.
In tensorflow/tensorflow/BUILD:598 there is a target:
tf_cc_shared_object(
name = "tensorflow_cc",
When I try to query it with bazel
bazel query //tensorflow:libtensorflow_cc --output location
I get this error:
ERROR: no such target '//tensorflow:libtensorflow_cc': target 'libtensorflow_cc' not declared in package 'tensorflow' (did you mean 'libtensorflow.so'?) defined by projects/tensorflow/tensorflow/BUILD
Why missing target refers other target, that I don't find in the BUILD file?
Shouldn't that be:
bazel query //tensorflow:tensorflow_cc --output location
which produces the following output for me:
[...]/tensorflow/tensorflow/BUILD:611:1: filegroup rule //tensorflow:tensorflow_cc
Loading: 0 packages loaded

bazel pypi macro self-edge

I'm stumbling around trying to get bazel working with pypi dependencies.
./pypi.bzl:
def _impl(ctx):
ctx.actions.run_shell(
command = "pip download %s" % ctx.package
)
_pypi_package = rule(
implementation=_impl,
attrs={"package": attr.label(mandatory=True)},
)
def pypi_package(package):
_pypi_package(name = package, package = package)
./BUILD:
py_binary(
name = "app",
srcs = ["app.py"],
deps = [":python-dateutil"]
)
load("//:pypi.bzl", "pypi_package")
pypi_package(
package="python-dateutil",
)
Trying to build:
$ bazel build app
ERROR: /path/to/cwd/BUILD:9:1: in _pypi_package rule //:python-dateutil: cycle in dependency graph:
//:app
.-> //:python-dateutil [self-edge]
`--
This cycle occurred because of a configuration option.
ERROR: Analysis of target '//:app' failed; build aborted.
INFO: Elapsed time: 0.219s
No idea if this is even the right approach for working with external dependencies, but ignoring that, I don't understand where the self-dep here is coming from. In fact, I don't see that I'm declaring any deps for the pypi_package rule at all. What's going on?
The issue was attr.label: "label" here means BUILD label. I should have been using attr.string.

"Genrules without outputs don't make sense" - Bazel Windows 10 build

I've been trying to compile tensorflow 1.1 inception with bazel 0.4.2 on windows 10 using CUDA 8.0.
I haven't been able to find a corresponding error online.
C:\Users\me\Anaconda3\envs\tensorflow_gpu\tensorflow>bazel build --config=opt tensorflow/examples/image_retraining:retrain
ERROR: C:/users/me/appdata/local/temp/_bazel_simon/qco1pmlq/external/local_config_cuda/cuda/BUILD:172:12: in outs attribute of genrule rule #local_config_cuda//cuda:cuda-include: Genrules without outputs don't make sense.
After digging into the Build file specified in the error message I have these genrules which are missing inputs:
genrule(
name = "cuda-include",
outs = [
],
cmd = """
""",
)
genrule(
name = "cuda-nvvm",
outs = [
],
cmd = """
""",
)
genrule(
name = "cuda-extras",
outs = [
],
cmd = """
""",
)
I am assuming these should have been generated? I do have an example of other rules that follow that format but unsure which files and if that is the correct way to go. Any help would be greatly appreciated.
Tensorflow GPU Build with Bazel on Windows is not very stable. Currently it's broken from both sides.
This change was just sent to fix problems in Bazel.
And this PR will make Cuda configuration work on Windows again.
You can first build Bazel from HEAD. And after the PR is merged, use your custom Bazel to build TensorFlow from HEAD.
The correct command on Windows would be:
bazel build -c opt --config=win-cuda --cpu=x64_windows_msvc --host_cpu=x64_windows_msvc --copt=-w --host_copt=-w tensorflow/tools/pip_package:build_pip_package
FYI, there is a script for building TensorFlow on Windows:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/ci_build/windows/gpu/pip/build_tf_windows.sh

Build chromium for raspberrypi2 on wayland with yocto project brings error related to egl.pc and wayland-egl.pc

I have completed building qt5 on raspberrypi2 by following the instruction here: http://www.embarcados.com.br/raspberry-pi-2-qt5-yocto-parte-1/
My next goal is to build chromium with wayland and egl. I have added one more layer from "https://github.com/OSSystems/meta-browser".
Actually, I could successfully build chromium with x11 but still failing with wayland. The things I did for wayland is to add "wayland" with DISTRO_FEATURES_append = " wayland" and remove x11 with DISTRO_FEATURES_remove = " x11".
However, this yields errors and the error messages mentions that missing "egl.pc" and "wayland-egl.pc". Should I add some more components in "DEPENDS" section which will be pre built and create those two package config files in addition to those components in chromium.inc? The build configurations and error messages shown are below.
Build Configuration:
BB_VERSION = "1.26.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "Ubuntu-14.04"
TARGET_SYS = "arm-poky-linux-gnueabi"
MACHINE = "raspberrypi2"
DISTRO = "poky"
DISTRO_VERSION = "1.8"
TUNE_FEATURES = "arm armv7a vfp thumb neon callconvention-hard vfpv4 cortexa7"
TARGET_FPU = "vfp-vfpv4-neon"
meta-embarcados
meta-rpi = "fido:dfb2e27bf26b109fc6ccb3bc91924fb3bb24f655"
meta-qt5 = "fido:fc026381545650e10fdd488d5b2a3b2d78f87793"
meta-raspberrypi = "fido:b896a7da70dd7a16ba7ffd664f7747cb37e1d142"
meta-oe
meta-ruby
meta-gnome = "fido:10d3c8f85280a0bf867a8e4f84bcda81c290d28e"
meta
meta-yocto = "fido:cd2c9acdbd75c83790e8144d2a834f5b5de35df0"
meta-browser = "master:ca96f4326345dcd779fc539877a04f069f431749"
NOTE: Preparing RunQueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: Function failed: do_configure (log file is located at /home/paul/yocto/build-browser-fido/tmp/work/cortexa7hf-vfp-vfpv4-neon-poky-linux-gnueabi/chromium/40.0.2214.91-r0/temp/log.do_configure.13805)
ERROR: Logfile of failure stored in: /home/paul/yocto/build-browser-fido/tmp/work/cortexa7hf-vfp-vfpv4-neon-poky-linux-gnueabi/chromium/40.0.2214.91-r0/temp/log.do_configure.13805
Updating projects from gyp files...
Package xcb-shm was not found in the pkg-config search path.
Perhaps you should add the directory containing xcb-shm.pc'
to the PKG_CONFIG_PATH environment variable
Package 'xcb-shm', required by 'cairo', not found
gyp: Call to 'pkg-config --cflags pangocairo pangoft2' returned exit status 1.
Requested 'egl >= 9.1.3' but version of egl is 0.0
Package wayland-egl was not found in the pkg-config search path.
Perhaps you should add the directory containingwayland-egl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'wayland-egl' found
gyp: Call to 'pkg-config --cflags "egl >= 9.1.3" "wayland-client >= 1.4.0" >>
Thank you very much for your help.
I found that I should have not done DISTRO_FEATURES_remove = " x11". However, I`m still attempting to launch chromium based on wayland compositor.