I tried to use ImageDataGenerator to build generator images to train my model, But I am unable to do so because of the PIL.UnidentifiedImageError error. I tried different datasets and the problem pertains only to my dataset.
Now I can't unfortunately delete all the training/testing images as an answer suggested but I can remove the files causing this problem. How can I detect the error causing files?
This is a common problem particularly if you download images from say google. I developed a function that given a directory, it will go through all sub directories and check the files in each sub directory to ensure the have proper extensions and are valid image files. Code is provided below. It returns two lists. good_list is a list of valid image files and bad_list is a list of invalid image files. You will need to have Opencv installed.If you do not have it installed use pip install opencv-contrib-python.
def test_images(dir):
import os
import cv2
bad_list=[]
good_list=[]
good_exts=['jpg', 'png', 'bmp','tiff','jpeg', 'gif'] # make a list of acceptable image file types
for klass in os.listdir(dir) : # iterate through the sub directories
class_path=os.path.join (dir, klass) # create path to sub directory
if os.path.isdir(class_path):
for f in os.listdir(class_path): # iterate through image files
f_path=os.path.join(class_path, f) # path to image files
ext=f[f.rfind('.')+1:] # get the files extension
if ext not in good_exts:
print(f'file {f_path} has an invalid extension {ext}')
bad_list.append(f_path)
else:
try:
img=cv2.imread(f_path)
size=img.shape
good_list.append(f_path)
except:
print(f'file {f_path} is not a valid image file ')
bad_list.append(f_path)
else:
print(f'** WARNING ** directory {dir} has files in it, it should only contain sub directories')
return good_list, bad_list
I am trying to do a basic colour conversion in python however I can't seem to get past the below error. I have re-installed python, opencv and tried on both python 3.4.3 (latest) and python 2.7 (which is on my Mac).
I installed opencv using python's package manager opencv-python.
Here is the code that fails:
frame = cv2.imread('frames/frame%d.tiff' % count)
frame_HSV= cv2.cvtColor(frame,cv2.COLOR_RGB2HSV)
This is the error message:
cv2.error: OpenCV(3.4.3) /Users/travis/build/skvark/opencv-python/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
This error happened because the image didn't load properly. So you have a problem with the previous line cv2.imread. My suggestion is :
check if the image exists in the path you give
check if the count variable has a valid number
If anyone is experiencing this same problem when reading a frame from a webcam:
Verify if your webcam is being used on another task and close it. This wil solve the problem.
I spent some time with this error when I realized my camera was online in a google hangouts group. Also, Make sure your webcam drivers are up to date
I kept getting this error too:
Traceback (most recent call last):
File "face_detector.py", line 6, in <module>
gray_img=cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\imgproc\src\color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor
My cv2.cvtColor(...) was working fine with \photo.jpg but not with \news.jpg. For me, I finally realized that when working on Windows with python, those escape characters will get you every time!! So my "bad" photo was being escaped because of the file name beginning with "n". Python took the \n as an escape character and OpenCV couldn't find the file!
Solution:
Preface file names in Windows python with r"...\...\" as in
cv2.imread(r".\images\news.jpg")
If the path is correct and the name of the image is OK, but you are still getting the error
use:
from skimage import io
img = io.imread(file_path)
instead of:
cv2.imread(file_path)
The function imread loads an image from the specified file and returns
it. If the image cannot be read (because of missing file, improper permissions, unsupported or invalid format), the function returns an empty matrix ( Mat::data==NULL ).
check if the image exists in the path and verify the image extension (.jpg or .png)
Check whether its the jpg, png, bmp file that you are providing and write the extension accordingly.
Another thing which might be causing this is a 'weird' symbol in your file and directory names. All umlaut (äöå) and other (éóâ etc) characters should be removed from the file and folder names. I've had this same issue sometimes because of these characters.
Most probably there is an error in loading the image, try checking directory again.
Print the image to confirm if it actually loaded or not
In my case, the image was incorrectly named. Check if the image exists and try
import numpy as np
import cv2
img = cv2.imread('image.png', 0)
cv2.imshow('image', img)
I've been in same situation as well, and My case was because of the Korean letter in the path...
After I remove Korean letters from the folder name, it works.
OR put
[#-*- coding:utf-8 -*-]
(except [ ] at the edge)
or something like that in the first line to make python understand Korean or your language or etc.
then it will work even if there is some Koreans in the path in my case.
So the things is, it seems like there is something about path or the letter.
People who answered are saying similar things. Hope you guys solve it!
I had the same problem and it turned out that my image names included special characters (e.g. château.jpg), which could not bet handled by cv2.imread. My solution was to make a temporary copy of the file, renaming it e.g. temp.jpg, which could be loaded by cv2.imread without any problems.
Note: I did not check the performance of shutil.copy2 vice versa other options. So probably there is a better/faster solution to make a temporary copy.
import shutil, sys, os, dlib, glob, cv2
for f in glob.glob(os.path.join(myfolder_path, "*.jpg")):
shutil.copy2(f, myfolder_path + 'temp.jpg')
img = cv2.imread(myfolder_path + 'temp.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
os.remove(myfolder_path + 'temp.jpg')
If there are only few files with special characters, renaming can also be done as an exeption, e.g.
for f in glob.glob(os.path.join(myfolder_path, "*.jpg")):
try:
img = cv2.imread(f)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
except:
shutil.copy2(f, myfolder_path + 'temp.jpg')
img = cv2.imread(myfolder_path + 'temp.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
os.remove(myfolder_path + 'temp.jpg')
In my case it was a permission issue. I had to:
chmod a+wrx the image,
then it worked.
must please see guys that the error is in the cv2.imread() .Give the right path of the image. and firstly, see if your system loads the image or not. this can be checked first by simple load of image using cv2.imread().
after that ,see this code for the face detection
import numpy as np
import cv2
cascPath = "/Users/mayurgupta/opt/anaconda3/lib/python3.7/site- packages/cv2/data/haarcascade_frontalface_default.xml"
eyePath = "/Users/mayurgupta/opt/anaconda3/lib/python3.7/site-packages/cv2/data/haarcascade_eye.xml"
smilePath = "/Users/mayurgupta/opt/anaconda3/lib/python3.7/site-packages/cv2/data/haarcascade_smile.xml"
face_cascade = cv2.CascadeClassifier(cascPath)
eye_cascade = cv2.CascadeClassifier(eyePath)
smile_cascade = cv2.CascadeClassifier(smilePath)
img = cv2.imread('WhatsApp Image 2020-04-04 at 8.43.18 PM.jpeg')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (x,y,w,h) in faces:
img = cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = img[y:y+h, x:x+w]
eyes = eye_cascade.detectMultiScale(roi_gray)
for (ex,ey,ew,eh) in eyes:
cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
cv2.imshow('img',img)
cv2.waitKey(0)
cv2.destroyAllWindows()
Here, cascPath ,eyePath ,smilePath should have the right actual path that's picked up from lib/python3.7/site-packages/cv2/data here this path should be to picked up the haarcascade files
Your code can't find the figure or the name of your figure named the by error message.
Solution:
import cv2
import numpy as np
import matplotlib.pyplot as plt
img=cv2.imread('哈哈.jpg')#solution:img=cv2.imread('haha.jpg')
print(img)
If anyone is experiencing this same problem when reading a frame from a webcam [with code similar to "frame = cv2.VideoCapture(0)"] and work in Jupyter Notebook, you may try:
ensure previously tried code is not running already and restart Jupyter Notebook kernel
SEPARATE code "frame = cv2.VideoCapture(0)" in separate cell on place where it is [previous code put in cell above, code under put to cell down]
then run all the code above cell where is "frame = cv2.VideoCapture(0)"
then try run next cell with its only code "frame = cv2.VideoCapture(0)" - AND - till you will continue in executing other cells - ENSURE - that ASTERIX on the left side of this particular cell DISAPEAR and command order number appear instead - only then continue
now you can try execute the rest of your code as your camera input should not be empty anymore :-)
After end, ensure you close all your program and restart kernel to prepare it for another run
As #shaked litbak , this error arised with my initial use with the ASCII-generator , as i naively thought i just had to add to the ./data directory , with its load automatically .
I had to append the --input option with the desired file path .
I checked my image file path and it was correct. I made sure there was no corrupt images.The problem was with my mac. It sometimes have a hidden file called .DS_Store which was saved together with the image file path. Therefore cv2 was having a problem with that file.So I solved the problem by deleting .DS_Store
I also encountered this type of error:
error: OpenCV(4.1.2) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
The solution was to load the image properly. Since the file mentioned was wrong, images were not loaded and hence it threw this error. You can check the path of the image or if uploading an image through colab or drive, make sure that the image is present in the drive.
I encounter the problem when I try to load the image from non-ASCII path.
If I simply use imread to load the image, I am only able to get None.
Here is my solution:
import cv2
import numpy as np
path = r'D:\map\上海地图\abc.png'
image = cv2.imdecode(np.fromfile(path, dtype=np.uint8), cv2.IMREAD_UNCHANGED)
Similar thing will happen when I save the image in a non-ASCII path. It will not be successfully saved without any warnings. And here is what I did.
import cv2
import numpy as np
path = r'D:\map\上海地图\abc.png'
cv2.imencode('.png', image)[1].tofile(path)
path = os.path.join(raw_folder, folder, file)
print('[DEBUG] path:', path)
img = cv2.imread(path) #read path Image
if img is None: # check if the image exists in the path you give
print('Wrong path:', path)
else: # It completes the steps
img = cv2.resize(img, dsize=(128,128))
pixels.append(img)
The solution os to ad './' before the name of image before reading it...
Just Try Degrading the OpenCV
in python Shell (in cmd)
>>> import cv2
>>> cv2.__version__
after Checking in cmd
pip uninstall opencv-python
after uninstalling the version of opencv install
pip install opencv-python==3.4.8.29
I am trying to work with the quite recently published tensorflow_dataset API to train a Keras model on the Open Images Dataset. The dataset is about 570 GB in size. I downloaded the data with the following code:
import tensorflow_datasets as tfds
import tensorflow as tf
open_images_dataset = tfds.image.OpenImagesV4()
open_images_dataset.download_and_prepare(download_dir="/notebooks/dataset/")
After the download was complete, the connection to my jupyter notebook somehow interrupted but the extraction seemed to be finished as well, at least all downloaded files had a counterpart in the "extracted" folder. However, I am not able to access the downloaded data now:
tfds.load(name="open_images_v4", data_dir="/notebooks/open_images_dataset/extracted/", download=False)
This only gives the following error:
AssertionError: Dataset open_images_v4: could not find data in /notebooks/open_images_dataset/extracted/. Please make sure to call dataset_builder.download_and_prepare(), or pass download=True to tfds.load() before trying to access the tf.data.Dataset object.
When I call the function download_and_prepare() it only downloads the whole dataset again.
Am I missing something here?
Edit:
After the download the folder under "extracted" has 18 .tar.gz files.
This is with tensorflow-datasets 1.0.1 and tensorflow 2.0.
The folder hierarchy should be like this:
/notebooks/open_images_dataset/extracted/open_images_v4/0.1.0
All the datasets have a version. Then the data could be loaded like this.
ds = tf.load('open_images_v4', data_dir='/notebooks/open_images_dataset/extracted', download=False)
I didn't have open_images_v4 data. I put cifar10 data into a folder named open_images_v4 to check what folder structure tensorflow_datasets was expecting.
The solution to this was to also use the "data_dir" parameter when initializing the dataset:
builder = tfds.image.OpenImagesV4(data_dir="/raid/openimages/dataset")
builder.download_and_prepare(download_dir="/raid/openimages/dataset")
This way the dataset is donwloaded and extracted in the same directory. Before, it was (for me unnoticeably) extracting to the default directory, which is under /home/.../. That's what caused the error, as there wasn't enough space left under my home directory.
After the extraction, the folder structure is exactly as Manoj-Mohan described.
Above solution haven't worked for me.
builder = tfds.builder(name='folder_name', data_dir=data_dir)
builder.download_and_prepare(download_dir="/home/...")
ds = builder.as_dataset()
I am trying to download data from Fashion MNIST, but it produces an error. Originally, it was downloading and working properly, but I had to terminate it because I had to turn off my computer. Once I opened the file up again, it gives me an error. I'm not sure what the problem is, but is it because I already downloaded some parts of the data once, and keras doesn't recognize that? I am using Jupyter notebook in a conda environment
Here is the link to the image:
https://i.stack.imgur.com/wLGDm.png
You have missed adding tf. to the line
fashion_mnist = keras.datasets.fashion_mnist
The below code works perfectly for me. Importing the fashion_mnist dataset has been outlined in tensorflow documention here.
Change your code to:
import tensorflow as tf
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
or, use the better way to do it below. This avoids creating an extra variable fashion_mnist:
import tensorflow as tf
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()
I am using tensorflow 1.9.0, keras 2.2.2 and python 3.6.6 on Windows 10 x64 OS.
I know my pc well, I can't download anything larger than 2.7 MB (in terminal), due to WinError 8.
So I manually downloaded all packs from storage.google (since some packs are 25 MB).
Check the packs:
then I paste all packs to \datasets\fashion-mnist
The next time u run your code, it should be fixed.
Note : If u have VScode then just CTRL and click the link, then you can download it easily.
I had an error regarding the cURL connection, and by looking into the error message I was able to track the file where the URL was declared. In my case it was:
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow_core/python/keras/datasets/fashion_mnist.py
At line 44 I have commented out the line:
# base = 'https://storage.googleapis.com/tensorflow/tf-keras-datasets/'
And declared a different base URL, which I had found looking into the documentation of the original dataset:
base = 'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/'
The download started immediately and gave no errors. Hope this helps.
This is because for some reason you have an incomplete download for the MNIST dataset.
You will have to manually delete the downloaded folder which usually resides in ~/.keras/datasets or any path specified by you relative to this path, in your case MNIST_data.
Go to : C:\Users\Username.keras\datasets
and then Delete the Dataset that you want to redownload or has the error
You should be good to go!
You can also manually add print for the path from which it is taking dataset ..
Ex: print(paths) in file fashion_mnist.py
with gzip.open(paths[3], 'rb') as imgpath:
print(paths) #debug print in fashion_mnist.py
x_test = np.frombuffer(
imgpath.read(), np.uint8, offset=16).reshape(len(y_test), 28, 28)
& from this path, remove the files & this will start to download fresh data ..
Change The base address with 'http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/' as described previously. It works for me.
I was getting error of Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
Traceback (most recent call last):
File "C:\Users\AsadA\AppData\Local\Programs\Python\Python38\lib\site-packages\numpy\lib\npyio.py", line 448, in load
return pickle.load(fid, **pickle_kwargs)
EOFError: Ran out of input
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\AsadA\AppData\Local\Programs\Python\Python38\lib\site-packages\numpy\lib\npyio.py", line 450, in load
raise IOError(
OSError: Failed to interpret file 'C:\\Users\\AsadA\\.keras\\datasets\\mnist.npz' as a pickle"**
GO TO FILE C:\Users\AsadA\AppData\Local\Programs\Python\Python38\Lib\site-packages\tensorflow\python\keras\datasets (In my Case) and follow the instructions: