Trying to save some experimental data to file I noticed that when trying to save NxN sized heatmaps the execution would never complete. Investigating further it appears to be due to the .pdf extension. If I use, for example, .png it's extremely fast.
Minimum reproducible example:
import matplotlib.pylab as plt
import numpy as np
import seaborn as sbn
N=200
THE_FIGURE = plt.figure(figsize=(8.27, 6), dpi=300)
ax = plt.subplot(1, 1, 1)
sbn.heatmap(np.random.uniform(1, 20, (N, N)), ax=ax)
THE_FIGURE.savefig('image.pdf', bbox_inches='tight', pad_inches=0.1)
This slowdown becomes noticeable even when N = 100.
N=1000 isn't even happening.
Is this normal? and how can I fix it
thanks!
It makes sense that for larger grids saving the pdf takes longer than saving the png. This can be seen in the following graph, where time for saving the pdf and png as a function of the number of tiles along one axis (N) is shown (solid lines). We can also look at the filesize of pdf and png, where some similar behaviour is oberved (dashed lines).
Find here the code for reproduction. Running this on my computer takes ~1:10 minutes.
import matplotlib.pylab as plt
import numpy as np
import seaborn as sns
import time
import os
def f(N, form = "pdf"):
t0= time.time()
fig = plt.figure(figsize=(8.27, 6), dpi=300)
ax = plt.subplot(1, 1, 1)
sns.heatmap(np.random.uniform(1, 20, (N, N)), ax=ax)
fig.savefig('image.'+form, bbox_inches='tight', pad_inches=0.1)
t1 = time.time()
plt.close(fig)
s = os.path.getsize('image.'+form)
return t1-t0,s
ns = [5,10,15,20,25,30] + range(40,210, 20)
pdf = []
png = []
for i,n in enumerate(ns):
pdf.append(f(n, form="pdf"))
png.append(f(n, form="png"))
#print i, n
pdf = np.array(pdf);png = np.array(png)
plt.figure()
plt.plot(ns, pdf[:,0], label="pdf")
plt.plot(ns, png[:,0], label="png")
plt.xlabel("N")
plt.ylabel("time [s]")
ax2 = plt.gca().twinx()
ax2.plot(ns, pdf[:,1]/1000., label="pdf (filesize)", ls="--")
ax2.plot(ns, png[:,1]/1000., label="png (filesize)", ls="--")
ax2.set_ylabel("filesize [kByte]")
plt.gcf().legend(ncol=2, loc="upper left", bbox_to_anchor=(0.125,0.98))
plt.subplots_adjust(top=0.85)
plt.show()
Also the reason seems intuitive. Png is a bitmap format, it saves the image as pixels. Pdf is a vector format, it saves the image as vector shapes.
While the png needs to store always the same amount of pixels (~2000x1500 in this case), it will take longer to save png for small N (here up to N=30 or NxN = 900). But the more tiles there are in the figure, the more shapes need to be stored in the pdf, hence it will eventually take longer to save many tiles in pdf format. We assume that the time it takes to save the pdf file is roughly proportionally to the amount of tiles to store. This suggests to have a quadratic relationship of time with N, time ~ N**2. Fitting a quadratic polynomial to the data and evaluating the polynomial at t=1000 gives
fit = np.polyfit(ns, pdf[:,0], 2)
print( np.poly1d(fit)(1000) )
gives 340 seconds, which is 5:40 minutes. This is the estimated time it takes to save the 1000x1000 matrix.
Note: All data here is produced on an Intel i5 3.5GHz windows computer running python 2.7 and matplotlib 2.1. Using a different computer will of course change the timings.
Related
I have troubles with a seemingly easy image processing task. I need to extract a curve from a medical image (the image I attached was fetched from an open source database).
Raw image
I also have about ~50 images like the attached one, but they have small differences and they vary a bit. I'm afraid it's not something that can be used with ML (due to small dataset)
I tried the following approach a modification of which is working well for blood vessels:
import os
import numpy as np
import cv2 as cv
from matplotlib import pyplot as plt
img = cv.imread('./img.png')
img_gray = img.copy()
img_gray[img_gray < 70] = 0
img_gray = cv.blur(img_gray * 1.5, (5, 5)).astype(np.uint8)
img_gray = 255 - img_gray
_, img_gray = cv.threshold(img_gray, 150, 255, cv.THRESH_BINARY_INV)
plt.axis('off')
plt.imshow(img_gray, cmap='gray')
plt.savefig('output.png', dpi=600, pad_inches=0, bbox_inches='tight')
Then we have the following result:
Processed curve
Any ideas of how to improve the filtering, please?
I'm doing some experiments with Scipy's STFT, and would like to confirm that I'm understanding things correctly.
The following code generates the image I would expect, but labeled with the wrong time values:
from math import ceil, log
from scipy.io.wavfile import read
from scipy.signal import stft
import numpy as np
import matplotlib.pyplot as plt
# read a 2s, 440 Hz test tone, padded with 0.5s of silence on either end
fs, x = read('a440_2s_padded.wav')
nperseg = 44100
# pick an FFT size that's the smallest power of 2 >= the window size
nfft = pow(2, ceil(log(nperseg, 2)))
# N.B. no overlap between windows
f, t, Zxx = stft(x, fs, 'blackman', nperseg=nperseg, noverlap=0, nfft=nfft, boundary='zeros')
# crop the display to relevant bins
minBin, maxBin = 600, 700
# plot it
plt.pcolormesh(t, f[minBin:maxBin], np.abs(Zxx[minBin:maxBin]), vmin=None, vmax=None)
plt.title('STFT Magnitude')
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [sec]')
plt.show()
matplotlib STFT output
As noted in the code, I'm analyzing a 2s, 440 Hz test tone, padded with 0.5s of silence on either end, but in the image, the signal starts at 1s and lasts until 3s. For small nperseg values, this discrepancy doesn't make much difference, but for large values and musical data, the difference can be substantial, as it determines whether the STFT is centering its frames within beats (the desired behavior), or on beats (undesired, because then it's smearing data from two consecutive beats).
Am I misunderstanding something about the STFT analysis settings? Thanks for any insight.
I can't seem to find a simple and fast way of plotting image sequences with plain matplotlib in a Jupyter Notebook. I've tried FuncAnimation, fig.canvas.draw(), blitting, as well as just the standard imshow-pause combo; without success or with very slow refresh rate. I don't need the images to be interactive - they just need to be shown sequentially and can't pop up a new figure window for each image. I've seen many solutions here, with none seeming to work the way I want.
My general pipeline does significant processing, with each image generated and plotted within a while or for loop. FuncAnimation is not desirable since it requires passing a function handle and my use case involves many arguments and state variables that make it difficult to use.
The best I've got is the working example below using fig.canvas.draw() - showing that drawing time increases linearly per iteration, where I need it to remain constant!
import numpy as np
import matplotlib.pyplot as plt
from timeit import default_timer as timer
%matplotlib notebook
num_iters = 50
im = np.arange(60).reshape((15,4))
fig, ax = plt.subplots(1,1)
fig.show()
fig.canvas.draw()
iter_times = np.zeros(num_iters)
for i in range(num_iters):
im = np.roll( a=im, shift=1, axis=0 )
t0 = timer()
ax.imshow(im.T, vmin=im.min(), vmax=im.max())
ax.set_title('Iter # {}/{}'.format(i+1, num_iters))
fig.canvas.draw()
iter_times[i] = timer()-t0
plt.figure(figsize=(6,3))
plt.plot(np.arange(num_iters)+1, iter_times)
plt.title('Imshow/drawing time per iteration')
plt.xlabel('Iteration number')
plt.ylabel('Time (seconds)')
plt.tight_layout()
plt.show()
I think the problem is that the plots are 'building up', so every one is being plotted every time. If you add ax.clear() right before the imshow(), you'll get linear plot times.
I have never worked with audio signals before and little do I know about signal processing. Nevertheless, I need to represent and audio signal using pyplot.specgram function from matplotlib library. Here is how I do it.
import matplotlib.pyplot as plt
import scipy.io.wavfile as wavfile
rate, frames = wavfile.read("song.wav")
plt.specgram(frames)
The result I am getting is this nice spectrogram below:
When I look at x-axis and y-axis which I suppose are frequency and time domains I can't get my head around the fact that frequency is scaled from 0 to 1.0 and time from 0 to 80k.
What is the intuition behind it and, what's more important, how to represent it in a human friendly format such that frequency is 0 to 100k and time is in sec?
As others have pointed out, you need to specify the sample rate, else you get a normalised frequency (between 0 and 1) and sample index (0 to 80k). Fortunately this is as simple as:
plt.specgram(frames, Fs=rate)
To expand on Nukolas answer and combining my Changing plot scale by a factor in matplotlib
and
matplotlib intelligent axis labels for timedelta
we can not only get kHz on the frequency axis, but also minutes and seconds on the time axis.
import matplotlib.pyplot as plt
import scipy.io.wavfile as wavfile
cmap = plt.get_cmap('viridis') # this may fail on older versions of matplotlib
vmin = -40 # hide anything below -40 dB
cmap.set_under(color='k', alpha=None)
rate, frames = wavfile.read("song.wav")
fig, ax = plt.subplots()
pxx, freq, t, cax = ax.specgram(frames[:, 0], # first channel
Fs=rate, # to get frequency axis in Hz
cmap=cmap, vmin=vmin)
cbar = fig.colorbar(cax)
cbar.set_label('Intensity dB')
ax.axis("tight")
# Prettify
import matplotlib
import datetime
ax.set_xlabel('time h:mm:ss')
ax.set_ylabel('frequency kHz')
scale = 1e3 # KHz
ticks = matplotlib.ticker.FuncFormatter(lambda x, pos: '{0:g}'.format(x/scale))
ax.yaxis.set_major_formatter(ticks)
def timeTicks(x, pos):
d = datetime.timedelta(seconds=x)
return str(d)
formatter = matplotlib.ticker.FuncFormatter(timeTicks)
ax.xaxis.set_major_formatter(formatter)
plt.show()
Result:
Firstly, a spectrogram is a representation of the spectral content of a signal as a function of time - this is a frequency-domain representation of the time-domain waveform (e.g. a sine wave, your file "song.wav" or some other arbitrary wave - that is, amplitude as a function of time).
The frequency values (y-axis, Hertz) are wholly dependant on the sampling frequency of your waveform ("song.wav") and will range from "0" to "sampling frequency / 2", with the upper limit being the "nyquist frequency" or "folding frequency" (https://en.wikipedia.org/wiki/Aliasing#Folding). The matplotlib specgram function will automatically determine the sampling frequency of the input waveform if it is not otherwise specified, which is defined as 1 / dt, with dt being the time interval between discrete samples of the waveform. You can can pass the option Fs='sampling rate' to the specgram function to manually define what it is. It will be easier for you to get your head around what is going on if you figure out and pass these variables to the specgram function yourself
The time values (x-axis, seconds) are purely dependent on the length of your "song.wav". You may notice some whitespace or padding if you use a large window length to calculate each spectra slice (think- the individual spectra which are arranged vertically and tiled horizontally to create the spectrogram image)
To make the axes more intuitive in the plot, use x- and y-axes labels and you can also scale the axes values (i.e. change the units) using a method similar to this
Take home message - try to be a bit more verbose with your code: see below for my example.
import matplotlib.pyplot as plt
import numpy as np
# generate a 5Hz sine wave
fs = 50
t = np.arange(0, 5, 1.0/fs)
f0 = 5
phi = np.pi/2
A = 1
x = A * np.sin(2 * np.pi * f0 * t +phi)
nfft = 25
# plot x-t, time-domain, i.e. source waveform
plt.subplot(211)
plt.plot(t, x)
plt.xlabel('time')
plt.ylabel('amplitude')
# plot power(f)-t, frequency-domain, i.e. spectrogram
plt.subplot(212)
# call specgram function, setting Fs (sampling frequency)
# and nfft (number of waveform samples, defining a time window,
# for which to compute the spectra)
plt.specgram(x, Fs=fs, NFFT=nfft, noverlap=5, detrend='mean', mode='psd')
plt.xlabel('time')
plt.ylabel('frequency')
plt.show()
5Hz_spectrogram:
I generated the attached image using matplotlib (png format). I would like to use eps or pdf, but I find that with all the data points, the figure is really slow to render on the screen. Other than just plotting less of the data, is there anyway to optimize it so that it loads faster?
I think you have three options:
As you mentioned yourself, you can plot fewer points. For the plot you showed in your question I think it would be fine to only plot every other point.
As #tcaswell stated in his comment, you can use a line instead of points which will be rendered more efficiently.
You could rasterize the blue dots. Matplotlib allows you to selectively rasterize single artists, so if you pass rasterized=True to the plotting command you will get a bitmapped version of the points in the output file. This will be way faster to load at the price of limited zooming due to the resolution of the bitmap. (Note that the axes and all the other elements of the plot will remain as vector graphics and font elements).
First, if you want to show a "trend" in your plot , and considering the x,y arrays you are plotting are "huge" you could apply a random sub-sampling to your x,y arrays, as a fraction of your data:
import numpy as np
import matplotlib.pyplot as plt
fraction = 0.50
x_resampled = []
y_resampled = []
for k in range(0,len(x)):
if np.random.rand() < fraction:
x_resampled.append(x[k])
y_resampled.append(y[k])
plt.scatter(x_resampled,y_resampled , s=6)
plt.show()
Second, have you considered using log-scale in the x-axis to increase visibility?
In this example, only the plotting area is rasterized, the axis are still in vector format:
import numpy as np
import matplotlib.pyplot as plt
x = np.random.uniform(size=400000)
y = np.random.uniform(size=400000)
plt.scatter(x, y, marker='x', rasterized=False)
plt.savefig("norm.pdf", format='pdf')