I have the following code taken from this tutorial.
def get_spectrogram(waveform):
zero_padding = tf.zeros([4900] - tf.shape(waveform), dtype=tf.float32)
waveform = tf.cast(waveform, tf.float32)
equal_length = tf.concat([waveform, zero_padding], 0)
spectrogram = tf.signal.stft(equal_length, frame_length=256, frame_step=128)
spectrogram = tf.abs(spectrogram)
return spectrogram
spectrogram = get_spectrogram(waveform)
print('Spectrogram shape:', spectrogram.shape)
And i have the following output of spectrogram shape.
Spectrogram shape: (37, 129)
What does the first and second value mean?
If I have 4900 samples and a frame_step of 128. Shouldn't the first value be 38?
4900/128 = 38.28125 -> 38 rounded
It also happens that with a Kotlin library I get a shape of (38, 127).
I need to understand, since I am implementing a model in Android with TFLite, therefore I am pre-processing the data from the mobile device.
I'm not familiar exactly with Python API, but assuming it's doing similar to WaveBeans which I'm very familiar with, it looks like what you've got is the 2-dimensional matrix.
What you're doing is a Short Fourier Transform, which is basically taking FFT over time. Whilst the FFT magnitude or phase is 2-dimensional and can be represented as a 1-dimensional vector, the SFT is 3-dimensional and have also the time axes, which is why it is 2-dimensional vector.
So it looks like the 38 side is time indexes, the 127 side is frequency index, the values are the FFT value on specific time-frequency bin, though that are complex numbers. Thinking of it as a polar coordinates, the phase is the angle, the magnitude is the length. In your code seems you're getting the magnitude by calling .abs() function, so you've already got rid of complex number representation.
Within WaveBeans there is an API to work with FFT specifically to extract out the phase and magnitude, as well as frequency values, and time values.
So to just keep the answer full I'll provide a code snippet:
// let's take simple sine as an example
val waveformAsAStream = 440.sine().trim(1000)
val fftStream = waveformAsAStream
.window(256,128)
// zero padding is already done inside, but if window.size == fft.size it doesn't really do anything
.fft(256)
// evaluate it, for example as a kotlin sequence
val stft = fftStream.asSequence(44100.0f)
.toList()
// get the specific sample for the sake of the demonstration
val fftSample = stft.drop(10).first()
// get time in nano seconds
fftSample.time()
// outputs the time of the taken sample:
// 29024943
// get frequencies values
fftSample.frequency().toList()
// outputs a list of size 128, each element is a frequency in Hz :
// [0.0, 172.265625, 344.53125, 516.796875, 689.0625, ..., 21360.9375, 21533.203125, 21705.46875, 21877.734375]
// get magnitude values
fftSample.magnitude().toList()
// outputs a list of size 128, each element is magnitude value for specific bin in dB:
// [29.629418039768613, 31.125367384785786, 38.077554502661705, 38.480916556622745, ..., -11.57802246867041]
// the index of the closest bin (index) of the frequency
fftSample.bin(440.0)
// outputs:
// 3
// get the magnitude in the FFT spectrogram of the specific frequency
fftSample.magnitude().toList()[fftSample.bin(440.0)]
// outputs:
// 38.480916556622745
Although I would recommend for better FFT output result to use window functions for example hamming is the popular one, and use less sized windows (zero padding will do the aligning trick in that case as FFT requires specific input length), i.e something like this:
waveformAsAStream
.window(101, 85)
.hamming()
.fft(256)
If you want to play around with the values you may use Kotlin Jupyter notebook with WaveBeans library, check it out on github
I need to find (and draw rect around)/get max and min radius blobs on the image. (samples below)
the problem is to find correct filters for the image that will allow Canny or Threshold transformation to highlight the blobs. then I going to use findContours to find the rectangles.
I've tryed:
Threshold - with different level
blur->erode->erode->grayscale->canny
change image tone with variety of "lines"
and ect. the better result was to detect piece (20-30%) of blob. and this info not allowed to draw rect around blob. also, thanks for shadows, not related to blob dots were detected, so that also prevents to detect the area.
as I understand I need to find counter that has hard contrast (not smooth like in shadow). Is there any way to do that with openCV?
Update
cases separately: image 1, image 2, image 3, image 4, image 5, image 6, image 7, image 8, image 9, image 10, image 11, image 12
One more Update
I believe that the blob have the contrast area at the edge. So, I've tried to make edge stronger: I've created 2 gray scale Mat: A and B, apply Gaussian blur for the second one - B (to reduce noise a bit), then I've made some calculations: goes around every pixel and find max difference between Xi,Yi of 'A' and nearby dots from 'B':
and apply max difference to Xi,Yi. so I get smth like this:
is i'm on the right way? btw, can I reach smth like this via OpenCV methods?
Update Image Denoising helps to reduce noize, Sobel - to highlight the contours, then threshold + findContours and custome convexHull gets smth similar I'm looking for but it not good for some blobs.
Since there are big differences between the input images, the algorithm should be able to adapt to the situation. Since Canny is based on detecting high frequencies, my algorithm treats the sharpness of the image as the parameter used for preprocessing adaptation. I didn't want to spend a week figuring out the functions for all the data, so I applied a simple, linear function based on 2 images and then tested with a third one. Here are my results:
Have in mind that this is a very basic approach and is only proving a point. It will need experiments, tests, and refining. The idea is to use Sobel and sum over all the pixels acquired. That, divided by the size of the image, should give you a basic estimation of high freq. response of the image. Now, experimentally, I found values of clipLimit for CLAHE filter that work in 2 test cases and found a linear function connecting the high freq. response of the input with a CLAHE filter, yielding good results.
sobel = get_sobel(img)
clip_limit = (-2.556) * np.sum(sobel)/(img.shape[0] * img.shape[1]) + 26.557
That's the adaptive part. Now for the contours. It took me a while to figure out a correct way of filtering out the noise. I settled for a simple trick: using contours finding twice. First I use it to filter out the unnecessary, noisy contours. Then I continue with some morphological magic to end up with correct blobs for the objects being detected (more details in the code). The final step is to filter bounding rectangles based on the calculated mean, since, on all of the samples, the blobs are of relatively similar size.
import cv2
import numpy as np
def unsharp_mask(img, blur_size = (5,5), imgWeight = 1.5, gaussianWeight = -0.5):
gaussian = cv2.GaussianBlur(img, (5,5), 0)
return cv2.addWeighted(img, imgWeight, gaussian, gaussianWeight, 0)
def smoother_edges(img, first_blur_size, second_blur_size = (5,5), imgWeight = 1.5, gaussianWeight = -0.5):
img = cv2.GaussianBlur(img, first_blur_size, 0)
return unsharp_mask(img, second_blur_size, imgWeight, gaussianWeight)
def close_image(img, size = (5,5)):
kernel = np.ones(size, np.uint8)
return cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)
def open_image(img, size = (5,5)):
kernel = np.ones(size, np.uint8)
return cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel)
def shrink_rect(rect, scale = 0.8):
center, (width, height), angle = rect
width = width * scale
height = height * scale
rect = center, (width, height), angle
return rect
def clahe(img, clip_limit = 2.0):
clahe = cv2.createCLAHE(clipLimit=clip_limit, tileGridSize=(5,5))
return clahe.apply(img)
def get_sobel(img, size = -1):
sobelx64f = cv2.Sobel(img,cv2.CV_64F,2,0,size)
abs_sobel64f = np.absolute(sobelx64f)
return np.uint8(abs_sobel64f)
img = cv2.imread("blobs4.jpg")
# save color copy for visualizing
imgc = img.copy()
# resize image to make the analytics easier (a form of filtering)
resize_times = 5
img = cv2.resize(img, None, img, fx = 1 / resize_times, fy = 1 / resize_times)
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# use sobel operator to evaluate high frequencies
sobel = get_sobel(img)
# experimentally calculated function - needs refining
clip_limit = (-2.556) * np.sum(sobel)/(img.shape[0] * img.shape[1]) + 26.557
# don't apply clahe if there is enough high freq to find blobs
if(clip_limit < 1.0):
clip_limit = 0.1
# limit clahe if there's not enough details - needs more tests
if(clip_limit > 8.0):
clip_limit = 8
# apply clahe and unsharp mask to improve high frequencies as much as possible
img = clahe(img, clip_limit)
img = unsharp_mask(img)
# filter the image to ensure edge continuity and perform Canny
# (values selected experimentally, using trackbars)
img_blurred = (cv2.GaussianBlur(img.copy(), (2*2+1,2*2+1), 0))
canny = cv2.Canny(img_blurred, 35, 95)
# find first contours
_, cnts, _ = cv2.findContours(canny.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
# prepare black image to draw contours
canvas = np.ones(img.shape, np.uint8)
for c in cnts:
l = cv2.arcLength(c, False)
x,y,w,h = cv2.boundingRect(c)
aspect_ratio = float(w)/h
# filter "bad" contours (values selected experimentally)
if l > 500:
continue
if l < 20:
continue
if aspect_ratio < 0.2:
continue
if aspect_ratio > 5:
continue
if l > 150 and (aspect_ratio > 10 or aspect_ratio < 0.1):
continue
# draw all the other contours
cv2.drawContours(canvas, [c], -1, (255, 255, 255), 2)
# perform closing and blurring, to close the gaps
canvas = close_image(canvas, (7,7))
img_blurred = cv2.GaussianBlur(canvas, (8*2+1,8*2+1), 0)
# smooth the edges a bit to make sure canny will find continuous edges
img_blurred = smoother_edges(img_blurred, (9,9))
kernel = np.ones((3,3), np.uint8)
# erode to make sure separate blobs are not touching each other
eroded = cv2.erode(img_blurred, kernel)
# perform necessary thresholding before Canny
_, im_th = cv2.threshold(eroded, 50, 255, cv2.THRESH_BINARY)
canny = cv2.Canny(im_th, 11, 33)
# find contours again. this time mostly the right ones
_, cnts, _ = cv2.findContours(canny.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# calculate the mean area of the contours' bounding rectangles
sum_area = 0
rect_list = []
for i,c in enumerate(cnts):
rect = cv2.minAreaRect(c)
_, (width, height), _ = rect
area = width*height
sum_area += area
rect_list.append(rect)
mean_area = sum_area / len(cnts)
# choose only rectangles that fulfill requirement:
# area > mean_area*0.6
for rect in rect_list:
_, (width, height), _ = rect
box = cv2.boxPoints(rect)
box = np.int0(box * 5)
area = width * height
if(area > mean_area*0.6):
# shrink the rectangles, since the shadows and reflections
# make the resulting rectangle a bit bigger
# the value was guessed - might need refinig
rect = shrink_rect(rect, 0.8)
box = cv2.boxPoints(rect)
box = np.int0(box * resize_times)
cv2.drawContours(imgc, [box], 0, (0,255,0),1)
# resize for visualizing purposes
imgc = cv2.resize(imgc, None, imgc, fx = 0.5, fy = 0.5)
cv2.imshow("imgc", imgc)
cv2.imwrite("result3.png", imgc)
cv2.waitKey(0)
Overall I think that's a very interesting problem, a little bit too big to be answered here. The approach I presented is due to be treated as a road sign, not a complete solution. Tha basic idea being:
Adaptive preprocessing.
Finding contours twice: for filtering and then for the actual classification.
Filtering the blobs based on their mean size.
Thanks for the fun and good luck!
Here is the code I used:
import cv2
from sympy import Point, Ellipse
import numpy as np
x1='C:\\Users\\Desktop\\python\\stack_over_flow\\XsXs9.png'
image = cv2.imread(x1,0)
image1 = cv2.imread(x1,1)
x,y=image.shape
median = cv2.GaussianBlur(image,(9,9),0)
median1 = cv2.GaussianBlur(image,(21,21),0)
a=median1-median
c=255-a
ret,thresh1 = cv2.threshold(c,12,255,cv2.THRESH_BINARY)
kernel=np.ones((5,5),np.uint8)
dilation = cv2.dilate(thresh1,kernel,iterations = 1)
kernel=np.ones((5,5),np.uint8)
opening = cv2.morphologyEx(dilation, cv2.MORPH_OPEN, kernel)
cv2.imwrite('D:\\test12345.jpg',opening)
ret,contours,hierarchy = cv2.findContours(opening,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
c=np.size(contours[:])
Blank_window=np.zeros([x,y,3])
Blank_window=np.uint8(Blank_window)
for u in range(0,c-1):
if (np.size(contours[u])>200):
ellipse = cv2.fitEllipse(contours[u])
(center,axes,orientation) =ellipse
majoraxis_length = max(axes)
minoraxis_length = min(axes)
eccentricity=(np.sqrt(1-(minoraxis_length/majoraxis_length)**2))
if (eccentricity<0.8):
cv2.drawContours(image1, contours, u, (255,1,255), 3)
cv2.imwrite('D:\\marked.jpg',image1)
Here problem is to find a near circular object. This simple solution is based on finding the eccentricity for each and every contour. Such objects being detected is the drop of water.
I have a partial solution in place.
FIRST
I initially converted the image to the HSV color space and tinkered with the value channel. On doing so I came across something unique. In almost every image, the droplets have a tiny reflection of light. This was highlighted distinctly in the value channel.
Upon inverting this I was able to obtain the following:
Sample 1:
Sample 2:
Sample 3:
SECOND
Now we have to extract the location of those points. To do so I performed anomaly detection on the inverted value channel obtained. By anomaly I mean the black dot present in them.
In order to do this I calculated the median of the inverted value channel. I allotted pixel value within 70% above and below the median to be treated as normal pixels. But every pixel value lying beyond this range to be anomalies. The black dots fit perfectly there.
Sample 1:
Sample 2:
Sample 3:
It did not turn out well for few images.
As you can see the black dot is due to the reflection of light which is unique to the droplets of water. Other circular edges might be present in the image but the reflection distinguishes the droplet from those edges.
THIRD
Now since we have the location of these black dots, we can perform Difference of Gaussians (DoG) (also mentioned in the update of the question) and obtain relevant edge information. If the obtained location of the black dots lie within the edges discovered it is said to be a water droplet.
Disclaimer: This method does not work for all the images. You can add your suggestions to this.
Good day , I am working on this subject and my advice to you is; First, after using many denoising filters such as Gaussian filters, process the image after that.
You can blob-detection these circles not with countors.
I can't not find the source of the difference in the handling of contrast for version 1.75.01 and 1.82. Here are two images that show what it used to look like (1.75),
and what it now looks like:
Unfortunately, rolling back is not trivial as I run into problems with dependencies (especially PIL v PILLOW). The images are created from a numpy array, and I suspect there is something related to how the numbers are getting handled (?type, rounding) when the conversion from array to image occurs, but I can't find the bug. Any help will be deeply appreciated.
Edited - New Minimal Example
#! /bin/bash
import numpy as np
from psychopy import visual,core
def makeRow (n,c):
cp = np.tile(c,[n,n,3])
cm = np.tile(-c,[n,n,3])
cpm = np.hstack((cp,cm))
return(cpm)
def makeCB (r1,r2,nr=99):
#nr is repeat number
(x,y,z) = r1.shape
if nr == 99:
nr = x/2
else:
hnr = nr/2
rr = np.vstack((r1,r2))
cb=np.tile(rr,[hnr,hnr/2,1])
return(cb)
def makeTarg(sqsz,targsz,con):
wr = makeRow(sqsz,1)
br = makeRow(sqsz,-1)
cb = makeCB(wr,br,targsz)
t = cb*con
return(t)
def main():
w = visual.Window(size = (400,400),units = "pix", winType = 'pyglet',colorSpace = 'rgb')
fullCon_np = makeTarg(8,8,1.0)
fullCon_i = visual.ImageStim(w, image = fullCon_np,size = fullCon_np.shape[0:2][::-1],pos = (-100,0),colorSpace = 'rgb')
fullCon_ih = visual.ImageStim(w, image = fullCon_np,size = fullCon_np.shape[0:2][::-1],pos = (-100,0),colorSpace = 'rgb')
fullCon_iz = visual.ImageStim(w, image = fullCon_np,size = fullCon_np.shape[0:2][::-1],pos = (-100,0),colorSpace = 'rgb')
fullCon_ih.contrast = 0.5
fullCon_ih.setPos((-100,100))
fullCon_iz.setPos((-100,-100))
fullCon_iz.contrast = 0.1
partCon_np = makeTarg(8,8,0.1)
partCon_i = visual.ImageStim(w, image = partCon_np,pos = (0,0), size = partCon_np.shape[0:2][::-1],colorSpace = 'rgb')
zeroCon_np = makeTarg(8,8,0.0)
zeroCon_i = visual.ImageStim(w, image = zeroCon_np,pos=(100,0), size = zeroCon_np.shape[0:2][::-1],colorSpace = 'rgb')
fullCon_i.draw()
partCon_i.draw()
fullCon_ih.draw()
fullCon_iz.draw()
zeroCon_i.draw()
w.flip()
core.wait(15)
core.quit()
if __name__ == "__main__":
main()
Which yields this:
The three checker-boards along the horizontal have the contrast changed in the array when generated before conversion to the image. The Vertical left shows that changing the image contrast afterwards works fine. The reason I can't use this is that a) I have collected a lot of data with the last version, and b) I want to grade the contrast of those big long bars in the centre programatically by multiplying one array against another, e.g. using a log scale or some other function, and doing the math is easier in numpy.
I still suspect the issue is in the conversion from np.array -> pil.image. The dtype of these array is float64, but even if I coerce to float32 nothing changes. If you examine the array before conversion at half contrast it is filled with 0.5 and -0.5 numbers, but all the negative numbers are getting turned to black and black is being set to zero at the time of conversion by psychopy.tools.imagetools.array2image I think.
OK, yes, the problem was to do with the issue of the scale for the array values. Basically, you've found a corner case that PsychoPy isn't handling correctly (i.e. a bug).
Explanation:
PsychoPy has a complex set of transformation rules for handling image/textures; it tries to deduce what you're going to do with this image and whether it should be stored in a way that supports colour manipulations (signed float) or not (can be an unsigned byte). In your case PsychoPy was getting this wrong; the fact that the array was filled with floats made PsychoPy think it could do color transforms, but the fact that it was NxNx3 suggest it shouldn't (we don't want to specify a "color" for something that already has its color specified for every pixel as rgb vals).
Workarounds (any one of these):
Just provide your array as NxN, not NxNx3. This is the right thing to do anyway; it means less for you to compute/store and by providing "intensity" values these can then be recolored on-the-fly. This is roughly what you had discovered already in providing just one slice of your NxNx3 array, but the point is that you could/should only create one slice in the first place.
Use GratingStim, which converts everything to signed floating point values rather than trying to work out what's best (potentially then you'd need to work out the spatial frequency stuff though)
You could add a line to fix it by rescaling your array (*0.5+0.5) but you'd have to set something so that this only occurred for this version (we'll fix it before the next release)
Basically, I'm suggesting you do (1) because that already works for past, present and future versions and is more efficient anyway. But thanks for letting us know - I'll try to make sure we catch this one in future
best wishes
Jon
The code is too long for me to read through and work out the actual problem.
What might be the problem is the issue of where zero should be. I think for a while numpy arrays were treated as having vals 0:1 whereas the rest of PsychoPy expects values to be -1:1 so it might be that you need to rescale your values with array=array*2-1 to get back to old (bad behaviour). Or check opacity too, which might have a similar issue. If you write a minimal example I'll read/test it properly
Thanks