bazel - Error building TensorFlow with local LLVM repository - tensorflow

Essentially, I want to run TensorFlow with a custom LLVM repository and not the llvm-mirror that bazel pulls from.
I made the following changes:
Changed the temp_workaround_http_archive rule in //tensorflow/workspace.bzl to:
native.local_repository (
name = "llvm",
path = "/git/llvm/",
)
In /git/llvm I added the file WORKSPACE containing:
workspace( name = "llvm" )
However, I know that an llvm.build file is required, but since I am new to bazel, I am not sure where it should be located.
I am getting the following error log:
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
ERROR: /git/tensorflow/tensorflow/tools/pip_package/BUILD:81:1: no such package '#llvm//': BUILD file not found on package path and referenced by '//tensorflow/tools/pip_package:licenses'.
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted.
INFO: Elapsed time: 0.219s
I installed TensorFlow from source. Here is the version info:
$ git rev-parse HEAD
4c3bb1aeb7bb46bea35036433742a720f39ce348
$ bazel version
Build label: 0.4.5
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Mar 16 12:19:38 2017 (1489666778)
Build timestamp: 1489666778
Build timestamp as int: 1489666778
Thanks in advance for the help!

Found the fix. Quite simple actually.
The local_repository rule in bazel is for external bazel repositories only. To use a non-bazel external repository, we need to use new_local_repository which takes build_file as an argument.

You can use the http server function of python to build a local file server, like:
python3 -m http.server
Then edit the file "tensorflow/workspace.bzl"
tf_http_archive(
name = "llvm",
urls = [
"https://mirror.bazel.build/**/195a164675af86f390f9816e53291013d1b551d7.tar.gz",
"http://localhost:8000/195a164675af86f390f9816e53291013d1b551d7.tar.gz",
"https://github.com/**/195a164675af86f390f9816e53291013d1b551d7.tar.gz",
],
sha256 = "57a8333f8e6095d49f1e597ca18e591aba8a89d417f4b58bceffc5fe1ffcc02b",
strip_prefix = "llvm-195a164675af86f390f9816e53291013d1b551d7",
build_file = str(Label("//third_party/llvm:llvm.BUILD")),
)
Add one local file path in the middle line of urls, and then rebuild it again.

Related

Problems with absl dependencies in TensorFlow print_selective_registration_header building

I'm trying to build //tensorflow/python/tools:print_selective_registration_header and get following output:
$ bazel build --copt="-DUSE_GEMM_FOR_CONV" tensorflow/python/tools:print_selective_registration_header
...
INFO: Analyzed target //tensorflow/python/tools:print_selective_registration_header (377 packages loaded, 24256 targets configured).
INFO: Found 1 target...
INFO: Deleting stale sandbox base /private/var/tmp/_bazel_glebdavydov/01d4cf1f69cd2606674d307a7a455e95/sandbox
ERROR: /Users/glebdavydov/Downloads/tensorflow-master/tensorflow/c/BUILD:372:1: undeclared inclusion(s) in rule '//tensorflow/c:ops':
this rule is missing dependency declarations for the following files included by 'tensorflow/c/ops.cc':
'tensorflow/contrib/makefile/downloads/absl/absl/strings/string_view.h'
'tensorflow/contrib/makefile/downloads/absl/absl/types/optional.h'
Target //tensorflow/python/tools:print_selective_registration_header failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 54.737s, Critical Path: 13.10s
INFO: 4 processes: 4 local.
FAILED: Build did NOT complete successfully
How can I fix this?
I fixed that issue by installing version of bazel corresponding to this build configurations.

Tensorflow bazel quantization build error

I am trying to build tensorflow tools package with bazel 0.18.0
following steps are ok
git clone https://github.com/tensorflow/tensoflow
bazel build --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel build --config=cuda //tensorflow/examples/label_image:label_image
until trying to run this command line
bazel build --config=cuda //tensorflow/contrib/quantization:quantize_graph
it show error, so should i give something else then quantize_graph? and what i can use or find?
root#24663fb1018d:/srv/wu/tensorflow-src/tensorflow# bazel build --config=cuda //tensorflow/contrib/quantization:quantize_graph
WARNING: Duplicate rc file: /srv/wu/tensorflow-src/tensorflow/tools/bazel.rc is read multiple times, most recently imported from /srv/wu/tensorflow-src/tensorflow/.bazelrc
WARNING: Processed legacy workspace file /srv/wu/tensorflow-src/tensorflow/tools/bazel.rc. This file will not be processedin the next release of Bazel. Please read https://github.com/bazelbuild/bazel/issues/6319 for further information, including how to upgrade.
Starting local Bazel server and connecting to it...
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
ERROR: Skipping '//tensorflow/contrib/quantization:quantize_graph': no such target '//tensorflow/contrib/quantization:quantize_graph': target 'quantize_graph' not declared in package 'tensorflow/contrib/quantization' defined by /srv/wu/tensorflow-src/tensorflow/tensorflow/contrib/quantization/BUILD
WARNING: Target pattern parsing failed.
ERROR: no such target '//tensorflow/contrib/quantization:quantize_graph': target 'quantize_graph' not declared in package 'tensorflow/contrib/quantization' defined by /srv/wu/tensorflow-src/tensorflow/tensorflow/contrib/quantization/BUILD
INFO: Elapsed time: 1.195s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded)
and then i tried with tools path, no luck
bazel build --config=cuda //tensorflow/tools/quantization:quantize_graph
WARNING: Duplicate rc file: /srv/wu/tensorflow-src/tensorflow/tools/bazel.rc is read multiple times, most recently imported from /srv/wu/tensorflow-src/tensorflow/.bazelrcWARNING: Processed legacy workspace file /srv/wu/tensorflow-src/tensorflow/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https://github.com/bazelbuild/bazel/issues/6319 for further information, including how to upgrade.WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.ERROR: Skipping '//tensorflow/tools/quantization:quantize_graph': no such package'tensorflow/tools/quantization': BUILD file not found on package pathWARNING: Target pattern parsing failed.
ERROR: no such package 'tensorflow/tools/quantization': BUILD file not found on package path
INFO: Elapsed time: 0.506s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
also the toco is not working
bazel build --config=cuda tensorflow/contrib/lite/toco:toco
WARNING: Duplicate rc file: /srv/wu/tensorflow-src/tensorflow/tools/bazel.rc is read multiple times, most recently imported from /srv/wu/tensorflow-src/tensorflow/.bazelrc
WARNING: Processed legacy workspace file /srv/wu/tensorflow-src/tensorflow/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https://github.com/bazelbuild/bazel/issues/6319 for further information, including how to upgrade.WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.ERROR: Skipping 'tensorflow/contrib/lite/toco:toco': no such package 'tensorflow/contrib/lite/toco': BUILD file not found on package path
WARNING: Target pattern parsing failed.ERROR: no such package 'tensorflow/contrib/lite/toco': BUILD file not found on package pathINFO: Elapsed time: 0.500s
INFO: 0 processes.FAILED: Build did NOT complete successfully (0 packages loaded)
To verify where targets are, look into the BUILD file in the package directory.
The quantize_graph target has been moved to the //tensorflow/contrib/quantize package. This should work:
$ bazel build --config=cuda //tensorflow/contrib/quantize:quantize_graph
The toco target has been moved from //tensorflow/contrib/lite/toco to //tensorflow/lite/toco. Like quantize_graph, this should work:
$ bazel build --config=cuda //tensorflow/lite/toco:toco

Compiling Tensorflow with Bazel

I tried compiling tensorflow 1.3 from the HEAD of the master branch using the following line of shell command after running ./configure
sudo bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 --config=cuda -k --verbose_failures //tensorflow/tools/pip_package:build_pip_package
I get the following error in the end.
At global scope:cc1plus: warning: unrecognized command line option '-Wno-self-assign'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 3834.785s, Critical Path: 196.95s FAILED: Build did NOT complete successfully
These were the warnings it gave initially.
WARNING: /home/pranav/tensorflow_install/tensorflow/tensorflow/core/BUILD:1634:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/nsync/public' resolves to 'external/nsync/public' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /home/pranav/tensorflow_install/tensorflow/tensorflow/tensorflow.bzl:911:30
WARNING: /home/pranav/tensorflow_install/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /home/pranav/tensorflow_install/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Analysed target //tensorflow/tools/pip_package:build_pip_package (208 packages loaded).
Then loads of INFO. I'm not sure if it is of any help.
Bazel Version:
Build label: 0.5.4
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Aug 25 10:00:00 2017 (1503655200)
Build timestamp: 1503655200
Build timestamp as int: 1503655200
I read in some answer to run the following code,
$ bazel query --output=build 'somepath("//tensorflow/core:version_info_gen", "//tensorflow/tools/git:gen/spec.json")'
And it gave me this.maybe this will be of help.
# /home/pranav/tensorflow_install/tensorflow/tensorflow/core/BUILD:1546:1
genrule(
name = "version_info_gen",
generator_name = "version_info_gen",
generator_function = "tf_version_info_genrule",
generator_location = "tensorflow/core/BUILD:1546",
srcs = ["//tensorflow/tools/git:gen/spec.json", "//tensorflow/tools/git:gen/head", "//tensorflow/tools/git:gen/branch_ref"],
tools = ["//tensorflow/tools/git:gen_git_source.py"],
outs = ["//tensorflow/core:util/version_info.cc"],
cmd = "$(location //tensorflow/tools/git:gen_git_source.py) --generate $(SRCS) \"$#\"",
local = True,
)
Also, "the bazel command i wrote" > log.txt doesn't fill the text file with the terminal outputs.
If you guys want more information to help me. Suggest me a way to copy the terminal output to a text file so that i can upload it on github and give you the link.
I also used --explain to write all explanations to a file . I can upload that also if you want.
I also tried --local_resources 2048,.5,1.0 to reduce my memory allocation in case of memory issues. Still doesn't work.
Thanks a lot in advance.

bazel pypi macro self-edge

I'm stumbling around trying to get bazel working with pypi dependencies.
./pypi.bzl:
def _impl(ctx):
ctx.actions.run_shell(
command = "pip download %s" % ctx.package
)
_pypi_package = rule(
implementation=_impl,
attrs={"package": attr.label(mandatory=True)},
)
def pypi_package(package):
_pypi_package(name = package, package = package)
./BUILD:
py_binary(
name = "app",
srcs = ["app.py"],
deps = [":python-dateutil"]
)
load("//:pypi.bzl", "pypi_package")
pypi_package(
package="python-dateutil",
)
Trying to build:
$ bazel build app
ERROR: /path/to/cwd/BUILD:9:1: in _pypi_package rule //:python-dateutil: cycle in dependency graph:
//:app
.-> //:python-dateutil [self-edge]
`--
This cycle occurred because of a configuration option.
ERROR: Analysis of target '//:app' failed; build aborted.
INFO: Elapsed time: 0.219s
No idea if this is even the right approach for working with external dependencies, but ignoring that, I don't understand where the self-dep here is coming from. In fact, I don't see that I'm declaring any deps for the pypi_package rule at all. What's going on?
The issue was attr.label: "label" here means BUILD label. I should have been using attr.string.

TensorFlow Serving Error

I have installed Tensorflow Serving as outlined on the install page at https://tensorflow.github.io/serving/setup. However, when I follow the build instruction on the page I get the following error:
$ bazel build tensorflow_serving/...
ERROR: /home/**PATH**/external/org_tensorflow/third_party/py/python_configure.bzl:183:20: unexpected keyword 'environ' in call to repository_rule(implementation: function, *, attrs: dict or NoneType = None, local: bool = False).
ERROR: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package '': Extension file 'third_party/py/python_configure.bzl' has errors.
INFO: Elapsed time: 0.623s
I am running on Ubuntu and TensorFlow 1.0.1 build. I am using Python 2.7 and have set up a virtualenv.
I can successfully build the bazel hello example and also am able to complete the gRPC quick start found at http://www.grpc.io/docs/quickstart/python.html.
Any suggestions?
-Dave
The trouble was an old copy of bazel. To determine your version
$ bazel version
Build label: 0.4.5
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Mar 16 12:19:38 2017 (1489666778)
Build timestamp: 1489666778
Build timestamp as int: 1489666778
In my case it required a manual removal of the old version
rm -fr ~/.bazel ~/.bazelrc
Next, I chose the install using the installer for ubuntu.
$ ./bazel-0.4.5-installer-linux-x86_64.sh
Bazel installer
---------------
Bazel is bundled with software licensed under the GPLv2 with Classpath exception.
You can find the sources next to the installer on our release page:
https://github.com/bazelbuild/bazel/releases
# Release 0.4.5 (2017-03-16)
There was still another trick to getting it to work.
$cd ..
$ bazel test tensorflow_serving/...
Python Configuration Error: 'PYTHON_BIN_PATH' environment variable is not set
This error is also related to versioning, but in this case it was an issue with serving. The solution was to revert to an earlier version and update the submodule from git (I had previously cloned the repository). From the serving directory:
$ git checkout 0.5.1
M tensorflow
M tf_models
Note: checking out '0.5.1'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 51bb356... Merge pull request #325 from kirilg/0.5.1
(tensorflow) $ git submodule update
Submodule path 'tensorflow': checked out '07bb8ea2379bd459832b23951fb20ec47f3fdbd4'
Submodule path 'tf_models': checked out '2fd3dcf3f31707820126a4d9ce595e6a1547385d'
(tensorflow) $ bazel test tensorflow_serving/...
Serving now reports success:
INFO: Found 199 targets and 57 test targets...
[1,299 / 4,037] Still waiting for 200 jobs to complete:
Running (standalone):