external/local_config_mlir/include/mlir/IR/Attributes.h:783:20: internal compiler error: in assign_temp, at function.c:968 - tensorflow

Question

I am trying to build Tensorflow 2.0 on Ubuntu 16.04LTS
I get the error with gcc 4.8, gcc 5, gcc 9.2, etc.
Here are the version of bazel and gcc:
ai) ubuntu#ip-10-0-1-71:~/tensorflow$ bazel --version
bazel 0.27.1
(ai) ubuntu#ip-10-0-1-71:~/tensorflow$ gcc --version
gcc (Ubuntu 9.2.1-17ubuntu1~16.04) 9.2.1 20191102
Here's the error message from gcc from bazel:
ERROR: /home/ubuntu/tensorflow/tensorflow/compiler/mlir/xla/BUILD:247:1: C++ compilation of rule '//tensorflow/compiler/mlir/xla:hlo' failed (Exit 1)
tensorflow/compiler/mlir/xla/ir/hlo_ops.cc: In function 'mlir::Type {anonymous}::GetBroadcastType(mlir::Builder*, mlir::Type, mlir::Type, mlir::Type, mlir::DenseIntElementsAttr)':
tensorflow/compiler/mlir/xla/ir/hlo_ops.cc:730:23: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int i = 0; i < shape_x.size(); i++) {
~~^~~~~~~~~~~~~~~~
In file included from tensorflow/compiler/mlir/xla/ir/hlo_ops.cc:113:0:
bazel-out/host/bin/tensorflow/compiler/mlir/xla/transforms/generated_canonicalize.inc: At global scope:
bazel-out/host/bin/tensorflow/compiler/mlir/xla/transforms/generated_canonicalize.inc:68:6: warning: 'void {anonymous}::populateWithGenerated(mlir::MLIRContext*, mlir::OwningRewritePatternList*)' defined but not used [-Wunused-function]
void populateWithGenerated(MLIRContext *context, OwningRewritePatternList *patterns) {
^~~~~~~~~~~~~~~~~~~~~
In file included from ./tensorflow/compiler/mlir/xla/ir/hlo_ops.h:22:0,
from tensorflow/compiler/mlir/xla/ir/hlo_ops.cc:18:
external/local_config_mlir/include/mlir/IR/Attributes.h: In member function 'T mlir::DenseElementsAttr::getValue(llvm::ArrayRef<long unsigned int>) const [with T = mlir::IntegerAttr]':
external/local_config_mlir/include/mlir/IR/Attributes.h:783:20: internal compiler error: in assign_temp, at function.c:968
auto castFn = [](Attribute attr) { return attr.template cast<T>(); };
^
I created an issue on github Tensoflow, but they haven't responded ...
https://github.com/tensorflow/tensorflow/issues/33431#issuecomment-543659836
with --verbose_failures:
ERROR: /home/ubuntu/tensorflow/tensorflow/compiler/mlir/xla/BUILD:247:1: C++ compilation of rule '//tensorflow/compiler/mlir/xla:hlo' failed (Exit 1): x86_64-conda_cos6-linux-gnu-cc failed: error executing command (cd /home/ubuntu/.cache/bazel/_bazel_ubuntu/ad1e09741bb4109fbc70ef8216b59ee2/execroot/org_tensorflow && \
exec env - \
LD_LIBRARY_PATH=/usr/local/cuda/lib64 \
PATH=/home/ubuntu/anaconda2/envs/ai/bin:/home/ubuntu/apps/maven/apache-maven-3.3.x-SNAPSHOT/bin:/home/ubuntu/anaconda2/bin:/usr/local/cuda-7.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/ubuntu/caffe/build/tools:/home/ubuntu/gnulib:.:.:/usr/lib/jvm/java-8-oracle/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin \
PWD=/proc/self/cwd \
PYTHONPATH=:/home/ubuntu/anaconda2/envs/ai/lib/python3.6/site-packages:/home/ubuntu/opencog/opencog/python/:/home/ubuntu/opencog/build/opencog/cython:/home/ubuntu/deepdist:/home/ubuntu/anaconda2/envs/ai/lib/python3.6/site-packages:/home/ubuntu/opencog/opencog/python/:/home/ubuntu/opencog/build/opencog/cython:/home/ubuntu/deepdist \
PYTHON_BIN_PATH=/home/ubuntu/anaconda2/envs/ai/bin/python \
PYTHON_LIB_PATH=/home/ubuntu/anaconda2/envs/ai/lib/python3.6/site-packages \
TF2_BEHAVIOR=1 \
TF_CONFIGURE_IOS=0 \
/home/ubuntu/anaconda2/envs/ai/bin/x86_64-conda_cos6-linux-gnu-cc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-py2-opt/bin/tensorflow/compiler/mlir/xla/_objs/hlo/hlo_ops.pic.d '-frandom-seed=bazel-out/k8-py2-opt/bin/tensorflow/compiler/mlir/xla/_objs/hlo/hlo_ops.pic.o' -fPIC -DLLVM_ENABLE_STATS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DLLVM_BUILD_GLOBAL_ISEL -iquote . -iquote bazel-out/k8-py2-opt/bin -iquote external/local_config_mlir -iquote bazel-out/k8-py2-opt/bin/external/local_config_mlir -iquote external/llvm -iquote bazel-out/k8-py2-opt/bin/external/llvm -iquote external/zlib_archive -iquote bazel-out/k8-py2-opt/bin/external/zlib_archive -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/CallOpInterfacesIncGen -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/DialectSymbolRegistry -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/InferTypeOpInterfaceIncGen -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/AffineOpsIncGen -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/LoopLikeOpInterfaceIncGen -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/StandardOpsIncGen -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/LoopOpsIncGen -Ibazel-out/k8-py2-opt/bin/external/local_config_mlir/_virtual_includes/VectorOpsIncGen -isystem tensorflow/compiler/mlir/xla/include -isystem bazel-out/k8-py2-opt/bin/tensorflow/compiler/mlir/xla/include -isystem external/local_config_mlir/include -isystem bazel-out/k8-py2-opt/bin/external/local_config_mlir/include -isystem external/llvm/include -isystem bazel-out/k8-py2-opt/bin/external/llvm/include -isystem external/zlib_archive -isystem bazel-out/k8-py2-opt/bin/external/zlib_archive '-march=native' -Wno-sign-compare '-std=c++14' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/compiler/mlir/xla/ir/hlo_ops.cc -o bazel-out/k8-py2-opt/bin/tensorflow/compiler/mlir/xla/_objs/hlo/hlo_ops.pic.o)
Execution platform: #bazel_tools//platforms:host_platform
[4,817 / 5,026] 3 actions running
In file included from tensorflow/compiler/mlir/xla/ir/hlo_ops.cc:113:0:
bazel-out/k8-py2-opt/bin/tensorflow/compiler/mlir/xla/transforms/generated_canonicalize.inc:68:6: warning: 'void {anonymous}::populateWithGenerated(mlir::MLIRContext*, mlir::OwningRewritePatternList*)' defined but not used [-Wunused-function]
void populateWithGenerated(MLIRContext *context, OwningRewritePatternList *patterns) {
^~~~~~~~~~~~~~~~~~~~~
In file included from ./tensorflow/compiler/mlir/xla/ir/hlo_ops.h:22:0,
from tensorflow/compiler/mlir/xla/ir/hlo_ops.cc:18:
external/local_config_mlir/include/mlir/IR/Attributes.h: In member function 'T mlir::DenseElementsAttr::getValue(llvm::ArrayRef<long unsigned int>) const [with T = mlir::IntegerAttr]':
external/local_config_mlir/include/mlir/IR/Attributes.h:783:20: internal compiler error: in assign_temp, at function.c:968
auto castFn = [](Attribute attr) { return attr.template cast<T>(); };
^
It appears to be using an Anaconda gcc:
ai) ubuntu#ip-10-0-1-71:~/tensorflow$ /home/ubuntu/anaconda2/envs/ai/bin/x86_64-conda_cos6-linux-gnu-cc --version
x86_64-conda_cos6-linux-gnu-cc (crosstool-NG 1.23.0.452-d158) 7.3.0
'bazel' is picking up 'gcc' from the GCC environment variable:
$ !env
env | grep GCC
GCC_NM=/usr/bin/nm
GCC_HOST_COMPILER_PREFIX=/usr/bin
GCC_RANLIB=/usr/bin/ranlib
GCC_HOST_COMPILER_PATH=/usr/bin/gcc
GCC=/usr/bin/gcc
GCC_AR=/usr/bin/ar
When I tried setting CC environment variable to:
GCC=/usr/bin/gcc
CC=/usr/bin/g++
I got the following errors:
/usr/bin/g++ -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/host/bin/external/zlib_archive/_objs/zlib/adler32.d '-frandom-seed=bazel-out/host/bin/external/zlib_archive/_objs/zlib/adler32.o' -iquote external/zlib_archive -iquote bazel-out/host/bin/external/zlib_archive -isystem external/zlib_archive -isystem bazel-out/host/bin/external/zlib_archive -g0 '-march=native' -Wno-shift-negative-value -DZ_HAVE_UNISTD_H -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/zlib_archive/adler32.c -o bazel-out/host/bin/external/zlib_archive/_objs/zlib/adler32.o)
Execution platform: #bazel_tools//platforms:host_platform
external/zlib_archive/adler32.c:63:25: error: 'uLong adler32_z' redeclared as different kind of symbol
uLong ZEXPORT adler32_z(adler, buf, len)
^~~~~
In file included from external/zlib_archive/zutil.h:22:0,
from external/zlib_archive/adler32.c:8:
external/zlib_archive/zlib.h:1707:23: note: previous declaration 'uLong adler32_z(uLong, const Bytef*, z_size_t)'
ZEXTERN uLong ZEXPORT adler32_z OF((uLong adler, const Bytef *buf,
^~~~~~~~~
external/zlib_archive/adler32.c:63:25: error: 'adler' was not declared in this scope
uLong ZEXPORT adler32_z(adler, buf, len)
^~~~~
external/zlib_archive/adler32.c:63:32: error: 'buf' was not declared in this scope
uLong ZEXPORT adler32_z(adler, buf, len)
^~~
external/zlib_archive/adler32.c:63:37: error: 'len' was not declared in this scope
uLong ZEXPORT adler32_z(adler, buf, len)
^~~
external/zlib_archive/adler32.c:67:1: error: expected unqualified-id before '{' token
{
^
external/zlib_archive/adler32.c:10:13: warning: 'uLong adler32_combine_(uLong, uLong, off64_t)' declared 'static' but never defined [-Wunused-function]
local uLong adler32_combine_ OF((uLong adler1, uLong adler2, z_off64_t len2));
^~~~~~~~~~~~~~~~
Any ideas?

Answer 1

This might happen if you have your config set such that warnings are errors.
Looking at the above it would appear that a compiler warning that is treated as a warning in TF OSS builds, your build is treating as an error.