While following Cloud TPU profiling guide and Bert FineTuning, I got error when creating Profile data.
Scalars and Graphs in TensorBoards are working well.
Is there anything I'm missing?
Configurations
Zone: us-central1-a(Both for Storage and TPU)
TPU Type: v3-8
TPU software version: tpu-vm-tf-2.7.0
TPU Architecture: TPU VM
Error log
Using CLI
(Run when training is process)
> capture_tpu_profile --tpu=bert-tpu --logdir=${MODEL_DIR} --duration_ms=3000
2022-01-20 06:34:29.301737: I tensorflow/core/tpu/tpu_initializer_helper.cc:68] libtpu.so already in used by another process. Not attempting to load libtpu.so in this process.
2022-01-20 06:34:29.301787: I tensorflow/core/tpu/tpu_api_dlsym_initializer.cc:116] Libtpu path is: libtpu.so
WARNING: Logging before InitGoogle() is written to STDERR
I0120 06:34:29.324573 67944 tpu_initializer_helper.cc:68] libtpu.so already in used by another process. Not attempting to load libtpu.so in this process.
2022-01-20 06:34:29.336671: I tensorflow/core/tpu/tpu_initializer_helper.cc:68] libtpu.so already in used by another process. Not attempting to load libtpu.so in this process.
2022-01-20 06:34:31.607899: I tensorflow/stream_executor/tpu/tpu_platform_interface.cc:77] No TPU platform registered. Waiting 1 second and trying again... (4 tries left)
2022-01-20 06:34:32.608170: I tensorflow/stream_executor/tpu/tpu_platform_interface.cc:77] No TPU platform registered. Waiting 1 second and trying again... (3 tries left)
2022-01-20 06:34:33.608461: I tensorflow/stream_executor/tpu/tpu_platform_interface.cc:77] No TPU platform registered. Waiting 1 second and trying again... (2 tries left)
2022-01-20 06:34:34.608757: I tensorflow/stream_executor/tpu/tpu_platform_interface.cc:77] No TPU platform registered. Waiting 1 second and trying again... (1 tries left)
2022-01-20 06:34:35.609050: I tensorflow/stream_executor/tpu/tpu_platform_interface.cc:74] No TPU platform found.
TensorFlow version 2.7.0 detected
Welcome to the Cloud TPU Profiler v2.4.0
I0120 06:34:35.628104 140127504198720 discovery.py:280] URL being requested: GET https://www.googleapis.com/discovery/v1/apis/tpu/v1/rest
I0120 06:34:35.709828 140127504198720 discovery.py:911] URL being requested: GET https://tpu.googleapis.com/v1/projects/elsa-270714/locations/us-central1-a/nodes/bert-tpu?alt=json
I0120 06:34:35.710047 140127504198720 transport.py:157] Attempting refresh to obtain initial access_token
I0120 06:34:35.710207 140127504198720 client.py:777] Refreshing access_token
I0120 06:34:35.806093 140127504198720 discovery.py:280] URL being requested: GET https://www.googleapis.com/discovery/v1/apis/tpu/v1/rest
I0120 06:34:35.838788 140127504198720 discovery.py:911] URL being requested: GET https://tpu.googleapis.com/v1/projects/elsa-270714/locations/us-central1-a/nodes/bert-tpu?alt=json
I0120 06:34:35.838929 140127504198720 transport.py:157] Attempting refresh to obtain initial access_token
I0120 06:34:35.839013 140127504198720 client.py:777] Refreshing access_token
Starting to trace for 3000 ms. Remaining attempt(s): 2
No trace event is collected. Automatically retrying.
Starting to trace for 3000 ms. Remaining attempt(s): 1
No trace event is collected. Automatically retrying.
Starting to trace for 3000 ms. Remaining attempt(s): 0
No trace event is collected after 3 attempt(s). Perhaps, you want to try again (with more attempts?).
Tip: increase number of attempts with --num_tracing_attempts.
Using TensorBoard
(TPU name: bert-tpu)
Packages
> pip3 list
Package Version
--------------------------------- --------------------
absl-py 0.12.0
anyio 3.5.0
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.0.5
astunparse 1.6.3
attrs 19.3.0
Automat 0.8.0
Babel 2.9.1
backcall 0.2.0
backports.entry-points-selectable 1.1.1
black 21.12b0
bleach 4.1.0
blinker 1.4
cachetools 4.2.4
certifi 2021.10.8
cffi 1.15.0
chardet 3.0.4
charset-normalizer 2.0.7
click 8.0.3
cloud-init 21.4
cloud-tpu-client 0.10
cloud-tpu-profiler 2.4.0
colorama 0.4.3
command-not-found 0.3
configobj 5.0.6
constantly 15.1.0
cryptography 2.8
cycler 0.11.0
Cython 0.29.24
dbus-python 1.2.16
debugpy 1.5.1
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.4
distlib 0.3.3
distro 1.4.0
distro-info 0.23ubuntu1
dm-tree 0.1.6
entrypoints 0.3
executing 0.8.2
filelock 3.4.0
flatbuffers 2.0
fonttools 4.28.5
future 0.18.2
gast 0.4.0
gin-config 0.5.0
google-api-core 1.31.4
google-api-python-client 1.8.0
google-auth 1.35.0
google-auth-httplib2 0.1.0
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
googleapis-common-protos 1.53.0
grpcio 1.42.0
gviz-api 1.10.0
h5py 3.6.0
httplib2 0.20.2
hyperlink 19.0.0
idna 3.3
importlib-metadata 4.8.2
importlib-resources 5.4.0
incremental 16.10.1
intel-openmp 2021.4.0
ipykernel 6.7.0
ipython 8.0.0
ipython-genutils 0.2.0
jax 0.2.25
jaxlib 0.1.74
jedi 0.18.1
Jinja2 2.10.1
joblib 1.1.0
json5 0.9.6
jsonpatch 1.22
jsonpointer 2.0
jsonschema 3.2.0
jupyter-client 7.1.1
jupyter-core 4.9.1
jupyter-server 1.13.3
jupyterlab 3.2.8
jupyterlab-pygments 0.1.2
jupyterlab-server 2.10.3
kaggle 1.5.12
keras 2.7.0
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.2
keyring 18.0.1
kiwisolver 1.3.2
language-selector 0.1
launchpadlib 1.10.13
lazr.restfulclient 0.14.2
lazr.uri 1.0.3
libclang 12.0.0
Markdown 3.3.6
MarkupSafe 1.1.0
matplotlib 3.5.1
matplotlib-inline 0.1.3
mistune 0.8.4
mkl 2021.4.0
mkl-include 2021.4.0
mock 4.0.3
more-itertools 4.2.0
mypy-extensions 0.4.3
nbclassic 0.3.5
nbclient 0.5.10
nbconvert 6.4.0
nbformat 5.1.3
nest-asyncio 1.5.4
netifaces 0.10.4
notebook 6.4.7
numpy 1.18.5
oauth2client 4.1.3
oauthlib 3.1.0
opencv-python-headless 4.5.5.62
opt-einsum 3.3.0
packaging 21.3
pandas 1.3.5
pandocfilters 1.5.0
parso 0.8.3
pathspec 0.9.0
pexpect 4.6.0
pickleshare 0.7.5
Pillow 8.4.0
pip 21.3.1
platformdirs 2.4.0
portalocker 2.3.2
prometheus-client 0.12.0
promise 2.3
prompt-toolkit 3.0.24
protobuf 3.19.1
psutil 5.9.0
ptyprocess 0.7.0
pure-eval 0.2.1
py-cpuinfo 8.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycocotools 2.0.4
pycparser 2.21
Pygments 2.11.2
PyGObject 3.36.0
PyHamcrest 1.9.0
PyJWT 1.7.1
pymacaroons 0.13.0
PyNaCl 1.3.0
pyOpenSSL 19.0.0
pyparsing 3.0.6
pyrsistent 0.15.5
pyserial 3.4
python-apt 2.0.0+ubuntu0.20.4.6
python-dateutil 2.8.2
python-debian 0.1.36ubuntu1
python-slugify 5.0.2
pytz 2021.3
PyYAML 5.4.1
pyzmq 22.3.0
regex 2022.1.18
requests 2.26.0
requests-oauthlib 1.3.0
requests-unixsocket 0.2.0
rsa 4.7.2
sacrebleu 2.0.0
scikit-learn 1.0.2
scipy 1.7.2
SecretStorage 2.3.1
Send2Trash 1.8.0
sentencepiece 0.1.96
seqeval 1.2.2
service-identity 18.1.0
setuptools 59.2.0
simplejson 3.16.0
six 1.16.0
sniffio 1.2.0
sos 4.1
ssh-import-id 5.10
stack-data 0.1.4
systemd-python 234
tabulate 0.8.9
tbb 2021.4.0
tensorboard 2.7.0
tensorboard-data-server 0.6.1
tensorboard-plugin-profile 2.5.0
tensorboard-plugin-wit 1.8.0
tensorflow 2.7.0
tensorflow-addons 0.15.0
tensorflow-datasets 4.4.0
tensorflow-estimator 2.7.0
tensorflow-hub 0.12.0
tensorflow-io-gcs-filesystem 0.22.0
tensorflow-metadata 1.5.0
tensorflow-model-optimization 0.7.0
tensorflow-text 2.7.0rc1
termcolor 1.1.0
terminado 0.12.1
testpath 0.5.0
text-unidecode 1.3
tf-slim 1.1.0
threadpoolctl 3.0.0
tomli 1.2.3
torch 1.11.0a0+git4635f57
torch-xla 1.11+73a3937
torchvision 0.12.0a0+59baae9
tornado 6.1
tqdm 4.62.3
traitlets 5.1.1
Twisted 18.9.0
typeguard 2.13.3
typing_extensions 4.0.0
ubuntu-advantage-tools 27.4
ufw 0.36
unattended-upgrades 0.1
uritemplate 3.0.1
urllib3 1.26.7
virtualenv 20.10.0
wadllib 1.3.3
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.2.3
Werkzeug 2.0.2
wheel 0.34.2
wrapt 1.13.3
zipp 3.7.0
zope.interface 4.7.1
Unfortunately capture_tpu_profile doesn't work with TPU VM.
If you're using TF2/Keras, one very accessible way is to use the TensorBoard Callback and set profile_batch=1 for instance. This would work for v3-8 but unfortunately wouldn't work for >v3-8.
Alternatively, you can use tf.profiler.experimental.start(...) and tf.profiler.experimental.stop() which is what the TensorBoard callback uses under the hood.
If you're using >v3-8 (for instance v3-32) you can use tf.profiler.experimental.client.trace() where service_addr is accessible from TPUClusterResolver's get_master() function.
Related
I created my anaconda environment and installed a few modules on it (see below full list) notably tensorflow and matplotlib
(FreeCodeCampML) C:\Users\abelm>conda list
# packages in environment at C:\Users\abelm\anaconda3\envs\FreeCodeCampML:
#
# Name Version Build Channel
_tflow_select 2.3.0 mkl
absl-py 1.4.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.1 py310he2412df_1 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
anyio 3.5.0 py310haa95532_0
appdirs 1.4.4 pyhd3eb1b0_0
argon2-cffi 21.3.0 pyhd3eb1b0_0
argon2-cffi-bindings 21.2.0 py310h2bbff1b_0
asttokens 2.0.5 pyhd3eb1b0_0
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
attrs 22.1.0 py310haa95532_0
babel 2.11.0 py310haa95532_0
backcall 0.2.0 pyhd3eb1b0_0
beautifulsoup4 4.11.1 py310haa95532_0
blas 1.0 mkl
bleach 4.1.0 pyhd3eb1b0_0
blinker 1.5 pyhd8ed1ab_0 conda-forge
bottleneck 1.3.5 py310h9128911_0
brotli 1.0.9 h2bbff1b_7
brotli-bin 1.0.9 h2bbff1b_7
brotlipy 0.7.0 py310h2bbff1b_1002
bzip2 1.0.8 he774522_0
ca-certificates 2022.12.7 h5b45459_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 5.3.0 pyhd8ed1ab_0 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h2bbff1b_3
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.1.3 win_pyhd8ed1ab_2 conda-forge
colorama 0.4.6 py310haa95532_0
comm 0.1.2 py310haa95532_0
cryptography 38.0.4 py310h21b164f_0
cycler 0.11.0 pyhd3eb1b0_0
debugpy 1.5.1 py310hd77b12b_0
decorator 5.1.1 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
entrypoints 0.4 py310haa95532_0
executing 0.8.3 pyhd3eb1b0_0
fftw 3.3.9 h2bbff1b_1
flatbuffers 2.0.0 h6c2663c_0
flit-core 3.6.0 pyhd3eb1b0_0
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 ha860e81_0
frozenlist 1.3.3 py310h2bbff1b_0
gast 0.4.0 pyh9f0ad1d_0 conda-forge
giflib 5.2.1 h8d14728_2 conda-forge
glib 2.69.1 h5dc1a3c_2
google-auth 2.16.0 pyh1a96a4e_1 conda-forge
google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
grpcio 1.42.0 py310hc60d5dd_0
gst-plugins-base 1.18.5 h9e645db_0
gstreamer 1.18.5 hd78058f_0
h5py 3.7.0 nompi_py310h00cbb18_100 conda-forge
hdf5 1.12.1 nompi_h2a0e4a3_100 conda-forge
icc_rt 2022.1.0 h6049295_2
icu 58.2 ha925a31_3
idna 3.4 py310haa95532_0
importlib-metadata 6.0.0 pyha770c72_0 conda-forge
intel-openmp 2021.4.0 haa95532_3556
ipykernel 6.19.2 py310h9909e9c_0
ipython 8.8.0 py310haa95532_0
ipython_genutils 0.2.0 pyhd3eb1b0_1
ipywidgets 7.6.5 pyhd3eb1b0_1
jedi 0.18.1 py310haa95532_1
jinja2 3.1.2 py310haa95532_0
jpeg 9e h2bbff1b_0
json5 0.9.6 pyhd3eb1b0_0
jsonschema 4.16.0 py310haa95532_0
jupyter 1.0.0 py310haa95532_8
jupyter_client 7.4.9 py310haa95532_0
jupyter_console 6.4.4 py310haa95532_0
jupyter_core 5.1.1 py310haa95532_0
jupyter_server 1.23.4 py310haa95532_0
jupyterlab 3.5.3 py310haa95532_0
jupyterlab_pygments 0.1.2 py_0
jupyterlab_server 2.16.5 py310haa95532_0
jupyterlab_widgets 1.0.0 pyhd3eb1b0_1
keras 2.10.0 py310haa95532_0 anaconda
keras-preprocessing 1.1.2 pyhd3eb1b0_0
kiwisolver 1.4.4 py310hd77b12b_0
lerc 3.0 hd77b12b_0
libbrotlicommon 1.0.9 h2bbff1b_7
libbrotlidec 1.0.9 h2bbff1b_7
libbrotlienc 1.0.9 h2bbff1b_7
libclang 12.0.0 default_h627e005_2
libcurl 7.87.0 h86230a5_0
libdeflate 1.8 h2bbff1b_5
libffi 3.4.2 hd77b12b_6
libiconv 1.16 h2bbff1b_2
libogg 1.3.5 h2bbff1b_1
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.20.3 h23ce68f_0
libsodium 1.0.18 h62dcd97_0
libssh2 1.10.0 h680486a_2 conda-forge
libtiff 4.5.0 h6c2663c_1
libvorbis 1.3.7 he774522_0
libwebp 1.2.4 h2bbff1b_0
libwebp-base 1.2.4 h2bbff1b_0
libxml2 2.9.14 h0ad7f3c_0
libxslt 1.1.35 h2bbff1b_0
lxml 4.9.1 py310h1985fb9_0
lz4-c 1.9.4 h2bbff1b_0
markdown 3.4.1 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.1 py310h2bbff1b_0
matplotlib 3.5.3 py310h5588dad_2 conda-forge
matplotlib-base 3.5.3 py310hd77b12b_0
matplotlib-inline 0.1.6 py310haa95532_0
mistune 0.8.4 py310h2bbff1b_1000
mkl 2021.4.0 haa95532_640
mkl-service 2.4.0 py310h2bbff1b_0
mkl_fft 1.3.1 py310ha0764ea_0
mkl_random 1.2.2 py310h4ed8f06_0
multidict 6.0.2 py310h2bbff1b_0
munkres 1.1.4 py_0
nbclassic 0.4.8 py310haa95532_0
nbclient 0.5.13 py310haa95532_0
nbconvert 6.5.4 py310haa95532_0
nbformat 5.7.0 py310haa95532_0
nest-asyncio 1.5.6 py310haa95532_0
notebook 6.5.2 py310haa95532_0
notebook-shim 0.2.2 py310haa95532_0
numexpr 2.8.4 py310hd213c9f_0
numpy 1.23.5 py310h60c9a35_0
numpy-base 1.23.5 py310h04254f7_0
oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge
openssl 1.1.1t h2bbff1b_0
opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge
packaging 22.0 py310haa95532_0
pandas 1.5.2 py310h4ed8f06_0
pandocfilters 1.5.0 pyhd3eb1b0_0
parso 0.8.3 pyhd3eb1b0_0
pcre 8.45 hd77b12b_0
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 9.3.0 py310hd77b12b_2
pip 22.3.1 py310haa95532_0
platformdirs 2.5.2 py310haa95532_0
ply 3.11 py310haa95532_0
pooch 1.4.0 pyhd3eb1b0_0
prometheus_client 0.14.1 py310haa95532_0
prompt-toolkit 3.0.36 py310haa95532_0
prompt_toolkit 3.0.36 hd3eb1b0_0
protobuf 3.20.3 py310hd77b12b_0
psutil 5.9.0 py310h2bbff1b_0
pure_eval 0.2.2 pyhd3eb1b0_0
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.21 pyhd3eb1b0_0
pygments 2.11.2 pyhd3eb1b0_0
pyjwt 2.6.0 pyhd8ed1ab_0 conda-forge
pyopenssl 22.0.0 pyhd3eb1b0_0
pyparsing 3.0.9 py310haa95532_0
pyqt 5.15.7 py310hd77b12b_0
pyqt5-sip 12.11.0 py310hd77b12b_0
pyrsistent 0.18.0 py310h2bbff1b_0
pysocks 1.7.1 py310haa95532_0
python 3.10.9 h966fe2a_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python-fastjsonschema 2.16.2 py310haa95532_0
python-flatbuffers 23.1.21 pyhd8ed1ab_0 conda-forge
python_abi 3.10 2_cp310 conda-forge
pytz 2022.7 py310haa95532_0
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pywin32 305 py310h2bbff1b_0
pywinpty 2.0.2 py310h5da7b33_0
pyzmq 23.2.0 py310hd77b12b_0
qt-main 5.15.2 he8e5bd7_7
qt-webengine 5.15.9 hb9a9bb5_5
qtconsole 5.4.0 pypi_0 pypi
qtpy 2.2.0 py310haa95532_0
qtwebkit 5.212 h3ad3cdb_4
requests 2.28.1 py310haa95532_0
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rsa 4.9 pyhd8ed1ab_0 conda-forge
scipy 1.10.0 py310hb9afe5d_0
send2trash 1.8.0 pyhd3eb1b0_1
setuptools 65.6.3 py310haa95532_0
sip 6.6.2 py310hd77b12b_0
six 1.16.0 pyhd3eb1b0_1
snappy 1.1.9 h82413e6_1 conda-forge
sniffio 1.2.0 py310haa95532_1
soupsieve 2.3.2.post1 py310haa95532_0
sqlite 3.40.1 h2bbff1b_0
stack_data 0.2.0 pyhd3eb1b0_0
tensorboard 2.10.0 py310haa95532_0
tensorboard-data-server 0.6.1 py310haa95532_0
tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge
tensorflow 2.10.0 mkl_py310hd99672f_0
tensorflow-base 2.10.0 mkl_py310h6a7f48e_0
tensorflow-estimator 2.10.0 py310haa95532_0
termcolor 2.2.0 pyhd8ed1ab_0 conda-forge
terminado 0.17.1 py310haa95532_0
tinycss2 1.2.1 py310haa95532_0
tk 8.6.12 h2bbff1b_0
toml 0.10.2 pyhd3eb1b0_0
tomli 2.0.1 py310haa95532_0
tornado 6.2 py310h2bbff1b_0
traitlets 5.7.1 py310haa95532_0
typing-extensions 4.4.0 py310haa95532_0
typing_extensions 4.4.0 py310haa95532_0
tzdata 2022g h04d1e81_0
urllib3 1.26.14 py310haa95532_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wcwidth 0.2.5 pyhd3eb1b0_0
webencodings 0.5.1 py310haa95532_1
websocket-client 0.58.0 py310haa95532_4
werkzeug 2.2.2 pyhd8ed1ab_0 conda-forge
wheel 0.37.1 pyhd3eb1b0_0
widgetsnbextension 3.5.2 py310haa95532_0
win_inet_pton 1.1.0 py310haa95532_0
wincertstore 0.2 py310haa95532_2
winpty 0.4.3 4
wrapt 1.14.1 py310he2412df_0 conda-forge
xz 5.2.10 h8cc25b3_1
yarl 1.7.2 py310he2412df_2 conda-forge
zeromq 4.3.4 hd77b12b_0
zipp 3.13.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h8cc25b3_0
zstd
When I run my code (see just below), I got the following error: "Canceled future for execute_request message before replies were done
The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further details."
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32, 32, 3))) #32 represents number of filters and (3,3) the size of the filters
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3),activation='relu'))
The first blocks of my code (which work fine) are as followed:
import tensorflow as tf
from tensorflow.keras import datasets, layers, models
import matplotlib.pyplot as plt
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
#Normalize pixel values to be between 0 and 1
train_images, test_images = train_images / 255, test_images
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
IMG_INDEX = 69
plt.imshow(train_images[IMG_INDEX], cmap=plt.cm.binary)
plt.xlabel(class_names[train_labels[IMG_INDEX][0]])
plt.show()
So I followed the instruction and tried to reinstall tensorflow. I also created a new environment to just have tensorflow and matplotlib as the modules I manually installed (fearing that other modules that I installed might interfere with the tensorflow one). I also used another environment with an older python version (3.9.16 instead of 3.10.9) Nothing worked
The instructions I followed come from github: "
If a kernel crashes when using tensorflow, this is indicative of tensorflow having been incorrectly installed into the Python Environment. Re-installing the package would resolve the issue.
If this does not work, it is also possible other dependent packages could cause the package to fall over, in such cases, its best to start out with a new environment.
Finally, when using Conda environments, please avoid using pip to install packages, instead use conda install.
Originally filed here https://github.com/microsoft/vscode-jupyter/issues/9283 and here https://github.com/microsoft/vscode-jupyter/issues/9157
Could you guys help ?
Before all manipulation, i created new virtual environment, and watch pip list, his have two modules:
pip 21.2.3
setuptools 57.4.0
Why, after this command:
pip install selenium
I watch this list modules?:
async-generator 1.10
attrs 21.4.0
certifi 2022.6.15
cffi 1.15.0
cryptography 37.0.2
h11 0.13.0
idna 3.3
outcome 1.2.0
pip 21.2.3
pycparser 2.21
pyOpenSSL 22.0.0
PySocks 1.7.1
selenium 4.2.0
setuptools 57.4.0
sniffio 1.2.0
sortedcontainers 2.4.0
trio 0.21.0
trio-websocket 0.9.2
urllib3 1.26.9
wsproto 1.1.0
I'ts normal? If True, please, send link to this info
I have set up and activated conda virtual environment that I use in Sublime Text 3. I have installed matplotlib into my conda virtual environment. When I try to generate a simple plot with the Conda build system, no plot is displayed and the code finishes running. I've tried editing the "Conda (Windows).sublime-settings" file to set "run_through_shell" to true but that hasn't fixed the problem. I've also tried adding "shell": true to the "Preferences.sublime-settings" but that hasn't worked either.
Edit: Matplotlib will plot when I import torch, but not when I don't have torch imported. Is there a dependency that comes along with torch that allows plots to be displayed?
Edit2: Here is the output of conda list for my virtual env:
# packages in environment at C:\Users\noami\anaconda3\envs\practice:
#
# Name Version Build Channel
blas 1.0 mkl
brotli 1.0.9 ha925a31_2
ca-certificates 2022.2.1 haa95532_0
certifi 2021.10.8 py39haa95532_2
cudatoolkit 10.2.89 h74a9793_1
cycler 0.11.0 pyhd3eb1b0_0
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.10.4 hd328e21_0
icu 58.2 ha925a31_3
intel-openmp 2021.4.0 haa95532_3556
jpeg 9d h2bbff1b_0
kiwisolver 1.3.2 py39hd77b12b_0
libpng 1.6.37 h2a8f88b_0
libtiff 4.2.0 hd0e1b90_0
libuv 1.40.0 he774522_0
libwebp 1.2.2 h2bbff1b_0
lz4-c 1.9.3 h2bbff1b_1
matplotlib 3.5.1 py39haa95532_0
matplotlib-base 3.5.1 py39hd77b12b_0
mkl 2021.4.0 haa95532_640
mkl-service 2.4.0 py39h2bbff1b_0
mkl_fft 1.3.1 py39h277e83a_0
mkl_random 1.2.2 py39hf11a4ad_0
munkres 1.1.4 py_0
numpy 1.21.5 py39ha4e8547_0
numpy-base 1.21.5 py39hc2deb75_0
olefile 0.46 pyhd3eb1b0_0
openssl 1.1.1m h2bbff1b_0
packaging 21.3 pyhd3eb1b0_0
pillow 8.4.0 py39hd45dc43_0
pip 21.2.4 py39haa95532_0
pyparsing 3.0.4 pyhd3eb1b0_0
pyqt 5.9.2 py39hd77b12b_6
python 3.9.7 h6244533_1
python-dateutil 2.8.2 pyhd3eb1b0_0
pytorch 1.10.2 py3.9_cuda10.2_cudnn7_0 pytorch
pytorch-mutex 1.0 cuda pytorch
qt 5.9.7 vc14h73c81de_0
setuptools 58.0.4 py39haa95532_0
sip 4.19.13 py39hd77b12b_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.37.2 h2bbff1b_0
tk 8.6.11 h2bbff1b_0
torchaudio 0.10.2 py39_cu102 pytorch
torchvision 0.11.3 py39_cu102 pytorch
tornado 6.1 py39h2bbff1b_0
typing_extensions 3.10.0.2 pyh06a4308_0
tzdata 2021e hda174b7_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py39haa95532_2
xz 5.2.5 h62dcd97_0
zlib 1.2.11 h8cc25b3_4
zstd 1.4.9 h19a0ad4_0
When I run this code, I don't get a plot:
import matplotlib.pyplot as plt
plt.plot([1,2,3,4,5])
plt.show()
print("Done!")
However when I run this code, I DO get a plot:
import matplotlib.pyplot as plt
import torch
plt.plot([1,2,3,4,5])
plt.show()
print("Done!")
I have been finished install Tensorflow env step by step from "https://developer.apple.com/metal/tensorflow-plugin/"
Tf is working!numpy is working! scipy is working!
but when i import sklearn package, have an error message like this:
ImportError: dlopen(/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so, 0x0002): Library not loaded: #rpath/liblapack.3.dylib
Referenced from: /Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so
Reason: tried: '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)
How to fix it?
pip list
Package Version
------------------------ -------------------
absl-py 0.10.0
aiohttp 3.8.1
aiosignal 1.2.0
anyio 3.5.0
appnope 0.1.2
argon2-cffi 20.1.0
astunparse 1.6.3
async-generator 1.10
async-timeout 4.0.1
attrs 21.4.0
Babel 2.9.1
backcall 0.2.0
beniget 0.3.0
bleach 4.1.0
blinker 1.4
Bottleneck 1.3.2
brotlipy 0.7.0
cached-property 1.5.2
cachetools 4.2.2
certifi 2021.10.8
cffi 1.15.0
charset-normalizer 2.0.4
click 8.0.3
cryptography 3.4.7
cycler 0.11.0
Cython 0.29.28
debugpy 1.5.1
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.4
entrypoints 0.3
flatbuffers 2.0
fonttools 4.25.0
frozenlist 1.2.0
gast 0.4.0
google-auth 1.33.0
google-auth-oauthlib 0.4.1
google-pasta 0.2.0
googleapis-common-protos 1.54.0
grpcio 1.42.0
h5py 3.1.0
idna 3.3
importlib-metadata 4.8.2
ipykernel 6.4.1
ipython 7.31.1
ipython-genutils 0.2.0
jedi 0.18.1
Jinja2 3.0.2
joblib 1.1.0
json5 0.9.6
jsonschema 3.2.0
jupyter-client 7.1.2
jupyter-core 4.9.1
jupyter-server 1.13.5
jupyterlab 3.2.1
jupyterlab-pygments 0.1.2
jupyterlab-server 2.10.3
keras 2.8.0
Keras-Preprocessing 1.1.2
kiwisolver 1.3.1
libclang 13.0.0
Markdown 3.3.4
MarkupSafe 2.0.1
matplotlib 3.5.0
matplotlib-inline 0.1.2
mistune 0.8.4
multidict 5.2.0
munkres 1.1.4
nbclassic 0.2.6
nbclient 0.5.3
nbconvert 6.3.0
nbformat 5.1.3
nest-asyncio 1.5.1
networkx 2.6.3
notebook 6.4.6
numexpr 2.8.1
numpy 1.22.2
oauthlib 3.1.1
opencv-python 4.5.5.62
opt-einsum 3.3.0
packaging 21.3
pandas 1.3.5
pandocfilters 1.5.0
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.0.1
pip 21.2.4
ply 3.11
prometheus-client 0.13.1
promise 2.3
prompt-toolkit 3.0.20
protobuf 3.19.1
ptyprocess 0.7.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pybind11 2.9.1
pycparser 2.21
Pygments 2.11.2
PyJWT 2.1.0
pyOpenSSL 21.0.0
pyparsing 3.0.4
pyrsistent 0.18.0
PySocks 1.7.1
python-dateutil 2.8.2
pythran 0.9.11
pytz 2021.3
pyzmq 22.3.0
requests 2.27.1
requests-oauthlib 1.3.0
rsa 4.7.2
scikit-learn 1.0.2
scipy 1.7.1
Send2Trash 1.8.0
setuptools 58.0.4
six 1.15.0
sklearn 0.0
sniffio 1.2.0
tensorboard 2.8.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.6.0
tensorflow-datasets 4.5.2
tensorflow-macos 2.8.0
tensorflow-metadata 1.6.0
tensorflow-metal 0.3.0
termcolor 1.1.0
terminado 0.13.1
testpath 0.5.0
tf-estimator-nightly 2.8.0.dev2021122109
threadpoolctl 2.2.0
tornado 6.1
tqdm 4.62.3
traitlets 5.1.1
typing-extensions 3.7.4.3
urllib3 1.26.8
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.58.0
Werkzeug 2.0.2
wheel 0.35.1
wrapt 1.12.1
yarl 1.6.3
zipp 3.7.0
You can install using Rosetta2 Mode.
To work in Rosetta Mode:
If Rosetta 2 is not installed by default in your M1 Mac, then open the pre-installed Terminal app and run the following command:
/usr/sbin/softwareupdate --install-rosetta --agree-to-license
Rosetta allows us to use apps built for Mac with intel chip.
Several CLI tools do not have native versions built for the new M1 architecture.
Enabling them on your native M1 Mac terminal can be frustrating.
Follow these steps to enable Rosetta:
Select the app(Terminal) in the Finder.
Right click on the app(Terminal) and select Get Info.
In General, check the Open using Rosetta check-box.
Close the Terminal Info.
Now when you quit the terminal and open it again.
If you haven't installed Rosetta yet, then it would prompt you to install it.
If the popup shows up, then click on Install button, then enter your user name and password to allow installation to proceed.
Close the Terminal and open again.
Now we have a special terminal that can install tools with Rosetta translation.
To verify that you are using a Rosetta terminal, run the following command and it should output i386:
arch
The native terminal without Rosetta would output arm64 for the above command.
Moving forward, all commands we ask you to execute should be done in Rosetta enabled terminal.
Uninstall arm64 brew
If you have installed brew in the past from the native terminal, it is likely that you have an arm64 build of brew. Having two different builds of brew can cause major problems as the packages with different builds will not be compatible with each other.
To avoid this problem you need to uninstall your current installation of arm64 brew.
You can check which build you have by running the following command:
which brew
If your installation of brew is the Intel build, then the command should output /usr/local/bin/brew. If that is the case you can skip installing brew and just update your current installation by running brew update.
If your output is /opt/homebrew then your installation of brew is the arm64 build.
You need to uninstall the arm64 build of brew by running the following command from the native terminal:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall.sh)"
Install Intel brew
Install Homebrew, which is the package manager:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Once done, run the below command to ensure that we make use of the HEAD revision:
git -C $(brew --repository homebrew/core) checkout master
Now verify the installation of the brew command:
which brew
The command should output /usr/local/bin/brew, which is the expected path.
Can we install tensorflow==0.11.0rc0 version in colab , as one of the pre-trained model code I use is coded in this version
You can install any version of TensorFlow in google collab.
However, there are specific versions that are available, so you may want to pick from those options . version 0.11.0rc0 is not currently available.
!pip install tensorflow==1.1.0rc0. #install a tensorflow version
import tensorflow as tf # import tensorflow
print(tf.__version__). # print tensorflow version
Here is the list of available versions as of now.
0.12.1, 1.0.0, 1.0.1, 1.1.0rc0, 1.1.0rc1, 1.1.0rc2, 1.1.0, 1.2.0rc0, 1.2.0rc1, 1.2.0rc2, 1.2.0, 1.2.1, 1.3.0rc0, 1.3.0rc1, 1.3.0rc2, 1.3.0, 1.4.0rc0, 1.4.0rc1, 1.4.0, 1.4.1, 1.5.0rc0, 1.5.0rc1, 1.5.0, 1.5.1, 1.6.0rc0, 1.6.0rc1, 1.6.0, 1.7.0rc0, 1.7.0rc1, 1.7.0, 1.7.1, 1.8.0rc0, 1.8.0rc1, 1.8.0, 1.9.0rc0, 1.9.0rc1, 1.9.0rc2, 1.9.0, 1.10.0rc0, 1.10.0rc1, 1.10.0, 1.10.1, 1.11.0rc0, 1.11.0rc1, 1.11.0rc2, 1.11.0, 1.12.0rc0, 1.12.0rc1, 1.12.0rc2, 1.12.0, 1.12.2, 1.12.3, 1.13.0rc0, 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 1.15.0rc0, 1.15.0rc1, 1.15.0rc2, 1.15.0rc3, 1.15.0, 1.15.2, 1.15.3, 2.0.0a0, 2.0.0b0, 2.0.0b1, 2.0.0rc0, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.0.1, 2.0.2, 2.1.0rc0, 2.1.0rc1, 2.1.0rc2, 2.1.0, 2.1.1, 2.2.0rc0, 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0