I am trying to follow this tutorial for using Yolov4 with transfer learning for a new object to detect: https://sandipanweb.wordpress.com/2022/01/17/custom-object-detection-with-transfer-learning-with-pre-trained-yolo-v4-model/. In trying to set up the initial file structure on Colab, I'm having to guess where files and folders are needed, and could use some help in identifying if this is the right file/folder structure. Or, if there is a more detailed tutorial or a way to understand what Yolov4 is going to need, that would be great.
There are several references to build/darknet/x64/data/. From what I can see when I explore the two downloads, this is really /content/build/darknet/x64/data/ in Colab. Is this a correct understanding?
Under the data folder, is Yolov4 looking for the train folder and the valid folder? That is, is it looking for /content/build/darknet/x64/data/train/ and /content/build/darknet/x64/data/valid/? I'm guessing the answer is yes.
Does Yolov4 need the _darknet.labels file along with all of the image and image annotation files. I am guessing yes because this is what's in the racoon dataset.
The build/darknet/x64/data/train.txt and build/darknet/x64/data/valid.txt files are to have the names of the images, so I'm guessing that name includes the .jpg extension because the tutorial specifically refers to images. The reason I question this is Yolov4 should also need the annotation file names, but that is not referenced in the tutorial. If Yolov4 strips the .jpg and adds the .txt to get the annotation file name that's great, but if it needs the file name w/o the extension so that it can add the extension to access both files, then I didn't understand that from this tutorial.
Any guidance would really be appreciated!
After working on this, here are some notes that have helped me get it to work:
After executing !git clone https://github.com/AlexeyAB/darknet/ change directories to /content/darknet before executing !wget -P build/darknet/x64/ https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.conv.137.
Although it may be obvious, to "change the Makefile to enable GPU and opencv", click on the icon on the left margin in Colab labeled "Files". In the darknet folder, open the Makefile (double click to open) and change the entries for GPU, CUDNN, and OPENCV from 0 to 1. ctrl-s to save the file, then make to compile the darknet executable.
To get around the path names referenced in the tutorial, I used full path names. Also, since the files referenced disappear when the Colab runtime is terminated, I copied the files from folders in Drive to local folders, some of which I needed to create. Here is the code I used:
!mkdir /content/darknet/build/darknet/x64/data/train
!mkdir /content/darknet/build/darknet/x64/data/valid
!cp /content/drive/MyDrive/your_folder_name/obj.data /content/darknet/build/darknet/x64/data/obj.data
!cp /content/drive/MyDrive/your_folder_name/obj.names /content/darknet/build/darknet/x64/data/obj.names
!cp /content/drive/MyDrive/your_folder_name_with_train_images/* /content/darknet/build/darknet/x64/data/train/
!cp /content/drive/MyDrive/your_folder_name_with_valid_images/* /content/darknet/build/darknet/x64/data/valid/
!cp /content/drive/MyDrive/your_folder_name/train.txt /content/darknet/build/darknet/x64/data/train.txt
!cp /content/drive/MyDrive/your_folder_name/valid.txt /content/darknet/build/darknet/x64/data/valid.txt
!cp /content/drive/MyDrive/your_folder_name/yolov4_train.cfg /content/darknet/build/darknet/x64/cfg/yolov4_train.cfg
In the folders with the images, I also included a _darknet.labels file (which just has the label for the single object you are detecting in it) and the annotation file (which has a zero in the first entry, then the bounding box coordinates using centerx, centery, width, heigth and just spaces in between--see the tutorial for an example). This was the answer for my question 3.
The train.txt and valid.txt files in my question 4 do work with just the .jpg file names.
The tutorial mentions that you need to change the number of filters and classes, but note this has to be changed in three locations.
Last, if you just want the prediction to print out in Colab for an image, this can be done by modifying /content/darknet/darknet.py before doing the make. In the detect_image function, insert a print(sorted(predictions, key=lambda x: x[1])) before the return. When the make is done, this will be picked up.
If you have not copied files in Colab, it requires some permissions to be set up in your first cells. Here are the two cells I run at the beginning (Colab will ask you to manually verify you are OK with access being granted when you run the cells):
`from google.colab import drive
drive.mount('/content/drive')`
`# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)`
Related
I tried to run a project (ipynb extension) from GitHub using Google colab.
I have managed to run the program, but when compared with the author’s output, mine is a little different.
For example, train_df.describe() does not print some of the columns (‘target' column in particular because that is used to plot a graph.
Why is it that I run the same program but get different result?
I was implementing a DCGAN application based on the lsun-bedroom dataset. I was planning to utilize tfds, since lsun was on its catalog. Since the total dataset contains 42.7 GB of images, I only wanted to load a portion(10%) of the full data and used the following code to load the data according to the manual. Unfortunately, the same error informing not enough disk space occurred. Would there be a possible solution with tfds or should I use another API to load the data?
tfds.load('lsun/bedroom',split='train[10%:]')
Not enough disk space. Needed: 42.77 GiB (download: 42.77 GiB, generated: Unknown size)
I was testing on Google Colab
TFDS download the dataset from the original author website. As the datasets are often published as monolithic archive (e.g lsun.zip), it is unfortunately
impossible for TFDS to only download/install part of the dataset.
The split argument only filter the dataset after it has been fully generated. Note: You can see the download size of the datasets in the catalog: https://www.tensorflow.org/datasets/catalog/overview
To me, there seems to be some kind of issue or, at least, a misunderstanding about the variable 'split' of tfds.load().
'split' seems to be intended to load a given portion of the dataset, once the whole dataset has been downloaded.
I got the same error message when downloading the dataset called "librispeech". Any setting of the variable 'split' seems to be intended to download the whole dataset, which is too big for my disk.
I managed to download the much smaller "mnist" dataset, but I found both the train and test splits downloaded by setting 'split' to 'test'.
During a model training, I got this error:
First I thought it was maybe a problem with my path, but it wasn't, and I discovered that a colab session does not support the symbolic link (ln) operation, for most of the ways you could run it (also with os.system):
My question is: Are there any other ways that I did not explore? or did someone get the same error with Colab? (not specifically with ln, but any other not supported operation)
Google Drive does not support symbolic link as it is a feature of Linux storage system. However, using the files directly in Colab, given that they can fit in, you can most definitely keep symbolic links. I would recommend compressing them and storing them if it's necessary to keep them in the drive.
You may have the wrong path.
Try changing from
/content/mydrive
To
/content/My Drive
Also check with
!ls "/content/My Drive/path/to/yourfile"
That the file really exists there.
Bridge is packaged with a script that will load multiple files as their own layer in a Photoshop file. There are two problems when you do this with a vector file:
It converts the files to raster layers. And since you don't get to choose the size of the file beforehand, if they're too small, you can't scale them up without losing quality.
It doesn't preserve antialiasing, leaving ugly jagged edges on whatever art you imported.
Is there a way to import multiple files into Photoshop as vector smart objects? Then you'd have full control over the quality. Alternatively, is there a way to define the size of the vector files you're loading into layers and/or preserve their antialiasing?
I found a script that loads files into Photoshop as smart objects, but this has the same two problems the factory Bridge script has. It appears to do the exact same thing, but converts the layers to smart objects after they are imported.
The only way I currently know of to get vector smart objects into Photoshop is to do so manually one by one by copying from Illustrator or by dragging the files to an open Photoshop file. I'm looking for a way to automate the process.
I'm afraid doing it manually is the only way to get where you want to go. I've wrestled with this same issue for years and hope with every PS/Bridge update they'll add the option to load a stack of smart objects, but so far it's still old-school drag n' drop.
Hit the Adobe suggestion box... maybe with enough requests they'll finally add this as a native feature.
I have few questions about TensorFlow. I'm following the "TensorFlow for Poets" tutorial (https://petewarden.com/2016/02/28/tensorflow-for-poets/), and i got the expected result.
However i would like to know two thing:
1. How to classify more than one image at a time?
2. How to extract the result in .txt format?
Thank you
I had the same issue, so I built the TensorPy GitHub repo to easily handle image classifications of either individual or multiple images directly from web pages. How it works: For multiple images, first it scrapes all the image links directly from a web page given. Then it downloads those images to a temporary folder and converts those images to JPEG format. Finally, it uses TensorFlow to classify all those images and print out the result, which you can then easily output to a txt file by adding " > output.txt" to the end of your command line run statement. See the video tutorial link in the repo, and swap out the individual image file from the example for a web page. Since you probably want your own customization, feel free to look at how the code works so that you can create your own version as you need it.
After creating my solution, I saw that there are also other good solutions available online. Check out Siraj's image classification tutorial, which has a link to the associated GitHub repo in the video description.
UPDATE:
If you're just looking to run TensorFlow's classify_image.py on multiple image files in a folder, you can easily create a bash script for that:
for i in temp_image_folder/*.jpg; do
python classify_image.py --image=$i
done
I am currently using the "find" command.
find ./ -type f -iname "*.jpg" -exec python3 classify_image.py --image={} \;
But I'm also looking for a solution that does not have to load the complete script for every image.