!kaggle competitions files -c planet-understanding-the-amazon-from-space
I ran the code in above, but it does not show me all the files. It only shows:
!kaggle competitions files -c planet-understanding-the-amazon-from-space
Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.6 /
client 1.5.4)
name size creationDate
------------------------------------------------- ----- -------------------
train_v2.csv/train_v2.csv 1MB 2019-12-15 22:14:13
sample_submission_v2.csv/sample_submission_v2.csv 3MB 2019-12-15 22:14:13
test_v2_file_mapping.csv/test_v2_file_mapping.csv 600KB 2019-12-15 22:14:13
Kaggle-planet-test-tif.torrent 2MB 2019-12-15 22:14:13
Kaggle-planet-train-tif.torrent 1MB 2019-12-15 22:14:13
It should show this, I can not download the train jpg tar file: all file image
[https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/data][2]
My colab: https://colab.research.google.com/drive/19hLo3NN_NY6tbXjEpWOI5y_bEVujikTF
Related
I am working with GPU supported VMs on GCP Compute Engine.
As OS I use a container optimized version (COS 89-16108.403.47 LTS), which supports simple GPU driver installation by running 'cos-extensions install gpu' via SSH (see Google doc).
This had worked perfectly so far until I started getting an error message saying that the download of some driver signature fails (see full error message below) a couple of days ago and I couldn't get it to work ever since.
Can someone either confirm that I am experiencing a bug here or help me fix this problem?
Many thanks in advance!
~ $ cos-extensions install gpu
Unable to find image 'gcr.io/cos-cloud/cos-gpu-installer:v2.0.3' locally
v2.0.3: Pulling from cos-cloud/cos-gpu-installer
419e7ae5bb1e: Pull complete
6f6ec2441524: Pull complete
11d24f918ba9: Pull complete
Digest: sha256:1cf2701dc2c3944a93fd06cb6c9eedfabf323425483ba3af294510621bb37d0e
Status: Downloaded newer image for gcr.io/cos-cloud/cos-gpu-installer:v2.0.3
I0618 06:33:49.227680 1502 main.go:21] Checking if this is the only cos_gpu_installer that is running.
I0618 06:33:49.258483 1502 install.go:74] Running on COS build id 16108.403.47
I0618 06:33:49.258505 1502 installer.go:187] Getting the default GPU driver version
I0618 06:33:49.285265 1502 utils.go:72] Downloading gpu_default_version from https://storage.googleapis.com/cos-
tools/16108.403.47/gpu_default_version
I0618 06:33:49.353149 1502 utils.go:120] Successfully downloaded gpu_default_version from https://storage.google
apis.com/cos-tools/16108.403.47/gpu_default_version
I0618 06:33:49.353381 1502 install.go:85] Installing GPU driver version 450.119.04
I0618 06:33:49.353461 1502 cache.go:69] error: failed to read file /root/var/lib/nvidia/.cache: open /root/var/l
ib/nvidia/.cache: no such file or directory
I0618 06:33:49.353482 1502 install.go:120] Did not find cached version, installing the drivers...
I0618 06:33:49.353491 1502 installer.go:82] Configuring driver installation directories
I0618 06:33:49.421021 1502 installer.go:196] Updating container's ld cache
I0618 06:33:49.526673 1502 signature.go:30] Downloading driver signature for version 450.119.04
I0618 06:33:49.526712 1502 utils.go:72] Downloading 450.119.04.signature.tar.gz from https://storage.googleapis.
com/cos-tools/16108.403.47/extensions/gpu/450.119.04.signature.tar.gz
E0618 06:33:49.657028 1502 artifacts.go:106] Failed to download extensions/gpu/450.119.04.signature.tar.gz from
public GCS: failed to download 450.119.04.signature.tar.gz, status: 404 Not Found
E0618 06:33:49.657487 1502 install.go:175] failed to download driver signature: failed to download driver signat
ure for version 450.119.04: failed to download extensions/gpu/450.119.04.signature.tar.gz
This seems to be a known issue, you can find it reported here and a similar thread with workarounds here.
Looks like there is a delay between the release of new COS version and release of updated drivers.
However, I ran cos-extensions list just now, and it seems there are drivers available:
$ cos-extensions list
Available extensions for COS version 89-16108.403.47:
[gpu]
450.119.04 [default]
450.80.02
And signatures as well:
$ wget https://storage.googleapis.com/cos-tools/16108.403.47/extensions/gpu/450.119.04.signature.tar.gz
--2021-06-21 12:49:58-- https://storage.googleapis.com/cos-tools/16108.403.47/extensions/gpu/450.119.04.signature.tar.gz
Resolving storage.googleapis.com... 173.194.198.128, 64.233.191.128, 173.194.74.128, ...
Connecting to storage.googleapis.com|173.194.198.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4588 (4.5K) [application/octet-stream]
Saving to: '450.119.04.signature.tar.gz'
450.119.04.signature.tar.gz 100%[=============================================>] 4.48K --.-KB/s in 0s
2021-06-21 12:49:58 (62.0 MB/s) - '450.119.04.signature.tar.gz' saved [4588/4588]
I am trying to create tfrecord (.record) files out of coco-format (.json). Therefore I am using this slightly modified version of the create_coco_tf_record.py script from the object detection api github repository.
I am running my notebook on colab. These are the few lines of code:
#Mount Google Drive.
from google.colab import drive
drive.mount('/content/gdrive') <br>
!pip install -U --pre tensorflow=="2.2.0"
#Download TensorFlow Model Garden.
import os
import pathlib
#cd into the TensorFlow directory in your Google Drive
%cd '/content/gdrive/My Drive/TensorFlow'
# Clone the tensorflow models repository if it doesn't already exist
if "models" in pathlib.Path.cwd().parts:
while "models" in pathlib.Path.cwd().parts:
os.chdir('..')
elif not pathlib.Path('models').exists():
!git clone --depth 1 https://github.com/tensorflow/models
# Install the Object Detection API
%%bash
cd '/content/gdrive/My Drive/TensorFlow/models/research/'
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .
#run model builder test
!python '/content/gdrive/My Drive/TensorFlow/models/research/object_detection/builders/model_builder_tf2_test.py'
!wget https://raw.githubusercontent.com/TannerGilbert/Tensorflow-Object-Detection-API-train-custom-Mask-R-CNN-model/master/create_coco_tf_record.py
# create tf record
!python create_coco_tf_record.py \
--logtostderr \
--train_image_dir='/content/gdrive/My Drive/TensorFlow/workspace/training_demo/images/combined/' \
--test_image_dir='/content/gdrive/My Drive/TensorFlow/workspace/training_demo/images/combined/' \
--train_annotations_file='/content/gdrive/My Drive/TensorFlow/workspace/training_demo/images/train_coco.json' \
--test_annotations_file='/content/gdrive/My Drive/TensorFlow/workspace/training_demo/images/test_coco.json' \
--output='/content/gdrive/My Drive/TensorFlow/workspace/training_demo/annotations/'
The create_coco_tf_record.py script runs through without any error. This is what it shows:
2020-10-28 08:58:44.931401: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
INFO:tensorflow:Found groundtruth annotations. Building annotations index.
I1028 08:58:46.678869 139783613331328 create_coco_tf_record.py:214] Found groundtruth annotations. Building annotations index.
INFO:tensorflow:0 images are missing annotations.
I1028 08:58:46.679595 139783613331328 create_coco_tf_record.py:227] 0 images are missing annotations.
INFO:tensorflow:On image 0 of 451
I1028 08:58:46.680609 139783613331328 create_coco_tf_record.py:232] On image 0 of 451
INFO:tensorflow:On image 100 of 451
I1028 08:58:51.800869 139783613331328 create_coco_tf_record.py:232] On image 100 of 451
INFO:tensorflow:On image 200 of 451
I1028 08:59:01.762672 139783613331328 create_coco_tf_record.py:232] On image 200 of 451
INFO:tensorflow:On image 300 of 451
I1028 08:59:22.197772 139783613331328 create_coco_tf_record.py:232] On image 300 of 451
INFO:tensorflow:On image 400 of 451
I1028 09:00:17.036898 139783613331328 create_coco_tf_record.py:232] On image 400 of 451
INFO:tensorflow:Finished writing, skipped 0 annotations.
I1028 09:00:32.919734 139783613331328 create_coco_tf_record.py:239] Finished writing, skipped 0 annotations.
INFO:tensorflow:Found groundtruth annotations. Building annotations index.
I1028 09:00:32.932144 139783613331328 create_coco_tf_record.py:214] Found groundtruth annotations. Building annotations index.
INFO:tensorflow:0 images are missing annotations.
I1028 09:00:32.932511 139783613331328 create_coco_tf_record.py:227] 0 images are missing annotations.
INFO:tensorflow:On image 0 of 152
I1028 09:00:32.932658 139783613331328 create_coco_tf_record.py:232] On image 0 of 152
INFO:tensorflow:On image 100 of 152
I1028 09:00:46.510094 139783613331328 create_coco_tf_record.py:232] On image 100 of 152
INFO:tensorflow:Finished writing, skipped 0 annotations.
I1028 09:01:08.650619 139783613331328 create_coco_tf_record.py:239] Finished writing, skipped 0 annotations.
Somehow it does not create any .record files.
Does anyone know what the problem could be here?
Thanks in advance!
You can run this command to generate your .record files in VS code terminal on a windows machine. "Imp. - Do not use 'commas' to include your path as I have mentioned just copy/paste path as such where I have suggested in the comment"-
python create_coco_tf_record.py
--train_image_dir 'Path where you have your training dataset'
--test_image_dir 'Path where you have your validation dataset'
--train_annotations_file 'Path to your training dataset .json file'
--test__annotations_file 'Path to your validation dataset .json file'
--logtostderr --output_dir 'Path where do you want to generate your .record file'
Also keep in mind and run the code with specific environment requirements needed to run this code.
I have this file with .7z prefix and I am trying to extract the content of it with google colab
I have tried this linux instruction below:
!7z img_celeba.7z
but unfortunately it give this error below:
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs Intel(R) Xeon(R) CPU # 2.30GHz (306F0),ASM,AES-NI)
Command Line Error:
Unsupported command:
img_celeba.7z
then I tried to install p7zip like below:
!apt-get install p7zip-full
!p7zip -d file_name.7z
I also get this error
7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,2 CPUs Intel(R) Xeon(R) CPU # 2.30GHz (306F0),ASM,AES-NI)
Scanning the drive for archives:
1 file, 734003200 bytes (700 MiB)
Extracting archive: img_celeba.7z
ERROR: img_celeba.7z
img_celeba.7z
Open ERROR: Can not open the file as [7z] archive
ERRORS:
Unexpected end of archive
Can't open as archive: 1
Files: 0
Size: 0
Compressed: 0
what should I do ??...
thanks in advance
I am going to close the question with instructions below:
first of all install p7zip using this comand:
!apt-get install p7zip-full
then try to extract the file with the below inst:
!p7zip -d file_name.7z
make sure that you provide the parent folder.7z
it worked for me it seems like I were tring to extract a part of the zipped folder
after some research it turn out the 7zipping partitioning the data in more than one section so all you need to do just refer to the right folder path with the prefix of .7z not its parts
tf-serving abnormal exit without error message
System information
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ReaHat EL6
TensorFlow Serving installed from (source or binary): source using bazel 0.18.0
TensorFlow Serving version: 1.12.0
Describe the problem
i compile the tf-serving using bazel in RHEL 6.9, and start it using:
./model_servers/tensorflow_model_server --model_config_file=./data/models.conf --rest_api_port=8502
models.conf:
model_config_list: {
config: {
name: "model_1",
base_path:"/search/work/tf_serving_bin/tensorflow_serving/data/model_data/model_1",
model_platform: "tensorflow",
model_version_policy: {
latest: {
num_versions: 1
}
}
}
}
Client using C++, and use libCurl to request tf-serving REST api, but, tf-serving often abnormal exits without error message in some minutes.
When my client service requests localhost tf-serving, the question occur frequently. But, client service requests tf-serving at other machines, the question do not occur, qps < 100.
I check memory, cpu idle, etc... no problems is found. so, it is very strange.
export export TF_CPP_MIN_VLOG_LEVEL=1, no error/critical message too.
Source code / logs
2019-01-09 09:28:35.118183: I tensorflow_serving/model_servers/server_core.cc:461] Adding/updating models.
2019-01-09 09:28:35.118259: I tensorflow_serving/model_servers/server_core.cc:558] (Re-)adding model: app_ks_nfm_1
2019-01-09 09:28:35.227383: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: app_ks_nfm_1 version: 201901072359}
2019-01-09 09:28:35.227424: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: app_ks_nfm_1 version: 201901072359}
2019-01-09 09:28:35.227443: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: app_ks_nfm_1 version: 201901072359}
2019-01-09 09:28:35.227492: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: /search/work/bazel-bin-serving/tensorflow_serving/data/model_data/app_ks_nfm_1/201901072359
2019-01-09 09:28:35.227530: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /search/work/bazel-bin-serving/tensorflow_serving/data/model_data/app_ks_nfm_1/201901072359
2019-01-09 09:28:35.256712: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-01-09 09:28:35.267728: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-01-09 09:28:35.313087: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:162] Restoring SavedModel bundle.
2019-01-09 09:28:38.797633: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:138] Running MainOp with key legacy_init_op on SavedModel bundle.
2019-01-09 09:28:38.803984: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:259] SavedModel load for tags { serve }; Status: success. Took 3570131 microseconds.
2019-01-09 09:28:38.804027: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:83] No warmup data file found at /search/work/bazel-bin-serving/tensorflow_serving/data/model_data/app_ks_nfm_1/201901072359/assets.extra/tf_serving_warmup_requests
2019-01-09 09:28:38.804148: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: app_ks_nfm_1 version: 201901072359}
2019-01-09 09:28:38.831860: I tensorflow_serving/model_servers/server.cc:286] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
2019-01-09 09:28:38.865243: I tensorflow_serving/model_servers/server.cc:302] Exporting HTTP/REST API at:localhost:8502 ...
[evhttp_server.cc : 237] RAW: Entering the event loop ...
It is not an abnormal exit. It is an indication that the Server is ready to receive the Inference Requests.
For clarification, please find the below explanation:
docker run --runtime=nvidia -p 8501:8501 \
--mount type=bind,\ source=/tmp/tfserving/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu,\
target=/models/half_plus_two \
-e MODEL_NAME=half_plus_two -t tensorflow/serving:latest-gpu &
This will run the docker container with the nvidia-docker runtime, launch the TensorFlow Serving Model Server, bind the REST API port 8501, and map our desired model from our host to where models are expected in the container. We also pass the name of the model as an environment variable, which will be important when we query the model.
TIP: Before querying the model, be sure to wait till you see a message like the following, indicating that the server is ready to receive requests:
2018-07-27 00:07:20.773693: I tensorflow_serving/model_servers/main.cc:333]
Exporting HTTP/REST API at:localhost:8501 ...
After that Message, just press Enter and you can query the model using the below command
curl -d '{"instances": [1.0, 2.0, 5.0]}' \
-X POST http://localhost:8501/v1/models/half_plus_two:predict
For more information, refer the below link:
https://www.tensorflow.org/tfx/serving/docker#gpu_serving_example
The Reason：the short connection product a large amount of TCP Status 'TIME_WAIT', the available linux system file handle is occupied.
I am getting 416 errors while creating buckets using S3 or Swift. How to solve this?
swift -A http://ceph-4:7480/auth/1.0 -U testuser:swift -K 'BKtVrq1...' upload testas testas
Warning: failed to create container 'testas': 416 Requested Range Not Satisfiable: InvalidRange
Object PUT failed: http://ceph-4:7480/swift/v1/testas/testas 404 Not Found b'NoSuchBucket'
Also S3 python test:
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 621, in create_bucket
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 416 Requested Range Not Satisfiable
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRange</Code><BucketName>mybucket</BucketName><RequestId>tx00000000000000000002a-005a69b12d-1195-default</RequestId><HostId>1195-default-default</HostId></Error>
Here is my ceph status:
cluster:
id: 1e4bd42a-7032-4f70-8d0c-d6417da85aa6
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph-2,ceph-3,ceph-4
mgr: ceph-1(active), standbys: ceph-2, ceph-3, ceph-4
osd: 3 osds: 3 up, 3 in
rgw: 2 daemons active
data:
pools: 7 pools, 296 pgs
objects: 333 objects, 373 MB
usage: 4398 MB used, 26309 MB / 30708 MB avail
pgs: 296 active+clean
I am using CEPH Luminous build with bluestore
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
User created:
sudo radosgw-admin user create --uid="testuser" --display-name="First User"
sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
Logs on osd:
2018-01-25 12:19:45.383298 7f03c77c4700 1 ====== starting new request req=0x7f03c77be1f0 =====
2018-01-25 12:19:47.711677 7f03c77c4700 1 ====== req done req=0x7f03c77be1f0 op status=-34 http_status=416 ======
2018-01-25 12:19:47.711937 7f03c77c4700 1 civetweb: 0x55bd9631d000: 192.168.109.47 - - [25/Jan/2018:12:19:45 +0200] "PUT /mybucket/ HTTP/1.1" 1 0 - Boto/2.38.0 Python/2.7.12 Linux/4.4.0-51-generic
Linux ubuntu, 4.4.0-51-generic
set default pg_num and pgp_num to lower value(8 for example), or set mon_max_pg_per_osd to a high value in ceph.conf