You may already know, that in matplotlib 1.2.0 there is a new experimental feature, that figures are pickable (they can be saved with pickle module).
However, it doesn't work when one uses logscale, eg.
import matplotlib.pyplot as plt
import numpy as np
import pickle
ax = plt.subplot(111)
x = np.linspace(0, 10)
y = np.exp(x)
plt.plot(x, y)
ax.set_yscale('log')
pickle.dump(ax, file('myplot.pickle', 'w'))
results in:
PicklingError: Can't pickle <class 'matplotlib.scale.Log10Transform'>: attribute lookup matplotlib.scale.Log10Transform failed
Anybody knows any solution/workaround to this?
I've opened this as a bug report on matplotlib's github issue tracker. Its a fairly easy fix to implement on the matplotlib repository side (simply don't nest the Log10Transform class inside the LogScale class), but that doesn't really help you in being able to use this with mpl 1.2.0...
There is a solution to getting this to work for you in 1.2.0, but I warn you - its not pretty!
Based on my answer to a pickling question it is possible to pickle nested classes (as Log10Transform is). All we need to do is to tell Log10Transform how to "reduce" itself:
import matplotlib.scale
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
the name of the nested class as the second argument,
and the state of the object as the third argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name, state):
nested_class = getattr(containing_class, class_name)
# return an instance of a nested_class. Some more intelligence could be
# applied for class construction if necessary.
c = nested_class.__new__(nested_class)
c.__setstate__(state)
return c
def _reduce(self):
# return a class which can return this class when called with the
# appropriate tuple of arguments
cls_name = matplotlib.scale.LogScale.Log10Transform.__name__
call_args = (matplotlib.scale.LogScale, cls_name, self.__getstate__())
return (_NestedClassGetter(), call_args)
matplotlib.scale.LogScale.Log10Transform.__reduce__ = _reduce
You might also decide to do this for other Log based transforms/classes, but for your example, you can now pickle (and successfully unpickle) your example figure:
import matplotlib.pyplot as plt
import numpy as np
import pickle
ax = plt.subplot(111)
x = np.linspace(0, 10)
y = np.exp(x)
plt.plot(x, y)
ax.set_yscale('log')
pickle.dump(ax, file('myplot.pickle', 'w'))
plt.savefig('pickle_log.pre.png')
plt.close()
pickle.load(file('myplot.pickle', 'r'))
plt.savefig('pickle_log.post.png')
I'm going to get on and fix this for mpl 1.3.x so that this nasty workaround isn't needed in the future :-) .
HTH,
Related
I have a dataset which I have to process in such a way that it works with a convolutional neural network of PyTorch (I'm completely new to PyTorch). The data is stored in a dataframe with a column for pictures (28 x 28 ndarrays with int32 entries) and a column with its class labels. The pixels of the images merely adopt values +1 and -1 (since it is simulation data of a classical 2d Ising Model). The dataframe looks like this.
I imported the following (a lot of this is not relevant for now, but I included everything for completeness. "data_loader" is a custom py file.):
import numpy as np
import matplotlib.pyplot as plt
import data_loader
import pandas as pd
import torch
import torchvision.transforms as T
from torchvision.utils import make_grid
from torch.nn import Module
from torch.nn import Conv2d
from torch.nn import Linear
from torch.nn import MaxPool2d
from torch.nn import ReLU
from torch.nn import LogSoftmax
from torch import flatten
from sklearn.metrics import classification_report
import time as time
from torch.utils.data import DataLoader, Dataset
Then, I want to get this in the correct shape in order to make it useful for PyTorch. I do this by defining the following class
class MetropolisDataset(Dataset):
def __init__(self, data_frame, transform=None):
self.data_frame = data_frame
self.transform = transform
def __len__(self):
return len(self.data_frame)
def __getitem__(self,idx):
if torch.is_tensor(idx):
idx = idx.tolist()
label = self.data_frame['label'].iloc[idx]
image = self.data_frame['image'].iloc[idx]
image = np.array(image)
if self.transform:
image = self.transform(image)
return (image, label)
I call instances of this class as:
train_set = MetropolisDataset(data_frame = df_train,
transform = T.Compose([
T.ToPILImage(),
T.ToTensor()]))
validation_set = MetropolisDataset(data_frame = df_validation,
transform = T.Compose([
T.ToPILImage(),
T.ToTensor()]))
test_set = MetropolisDataset(data_frame = df_test,
transform = T.Compose([
T.ToPILImage(),
T.ToTensor()]))
The problem does not yet arise here, because I am able to read out and show images from these instances of the above defined class.
Then, as far as I found out, it is necessary to let this go through the DataLoader of PyTorch, which I do as follows:
batch_size = 64
train_dl = DataLoader(train_set, batch_size, shuffle=True, num_workers=3, pin_memory=True)
validation_dl = DataLoader(validation_set, batch_size, shuffle=True, num_workers=3, pin_memory=True)
test_dl = DataLoader(test_set, batch_size, shuffle=True, num_workers=3, pin_memory=True)
However, if I want to use these instances of the DataLoader, simply nothing happens. I neither get an error, nor the computation seems to get anywhere. I tried to run a CNN but it does not seem to compute anything. Something else I tried was to show some sample images with the code provided by this article, but the same issue occurs. The sample code is:
def show_images(images, nmax=10):
fig, ax = plt.subplots(figsize=(8, 8))
ax.set_xticks([]); ax.set_yticks([])
ax.imshow(make_grid((images.detach()[:nmax]), nrow=8).permute(1, 2, 0))
def show_batch(dl, nmax=64):
for images in dl:
show_images(images, nmax)
break
show_batch(test_dl)
It seems that there is some error in the implementation of my MetropolisDataset class or with the DataLoader itself. How could this problem be solved?
As mentioned in the comments, the problem was partly solved by setting num_workers to zero since I was working in a Jupyter notebook, as answered here. However, this left open one further problem that I got errors when I wanted to apply the DataLoader to run a CNN. The issue was then that my data did consist of int32 numbers instead of float32. I do not include further codes, because this was related directly to my data - however, the issue was (as very often) merely a wrong datatype.
I got the following error:
<lambdifygenerated-1>:2: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.return numpy.array((A1exp(-1/2(x - xc1)**2/sigma1**2), 0, 0))
Here I have just one model but this code is written for model combination in fitting by the lmfit Please kindly let me know about it.
import matplotlib.pyplot as plt
import numpy as np
import sympy
from sympy.parsing import sympy_parser
import lmfit
gauss_peak1 = sympy_parser.parse_expr('A1*exp(-(x-xc1)**2/(2*sigma1**2))')
gauss_peak2 = 0
exp_back = 0
model_list = sympy.Array((gauss_peak1, gauss_peak2, exp_back))
model = sum(model_list)
print(model)
model_list_func = sympy.lambdify(list(model_list.free_symbols), model_list)
model_func = sympy.lambdify(list(model.free_symbols), model)
np.random.seed(1)
x = np.linspace(0, 10, 40)
param_values = dict(x=x, A1=2, sigma1=1, xc1=2)
y = model_func(**param_values)
yi = model_list_func(**param_values)
yn = y + np.random.randn(y.size)*0.4
plt.plot(x, yn, 'o')
plt.plot(x, y)
lm_mod = lmfit.Model(model_func, independent_vars=('x'))
res = lm_mod.fit(data=yn, **param_values)
res.plot_fit()
plt.plot(x, y, label='true')
plt.legend()
plt.show()
lmfit.Model takes a model function that is a Python function. It parses the function arguments and expects those to be the Parameters for the model.
I don't think using sympy-created functions will do that. Do you need to use sympy here? I don't see why. The usage here seems designed to make the code more complex than it needs to be. It seems you want to make a model with a Gaussian-like peak, and a constant(?) background. If so, why not do
from lmfit.Models import GaussianModel, ConstantModel
model = GaussianModel(prefix='p1_') + ConstantModel()
params = model.make_params(p1_amplitude=2, p1_center=2, p1_sigma=1, c=0)
That just seems way easier to me, and it is very easy to add a second Gaussian peak to that model.
But even if you have your own preferred mathematical expression, don't use that as a sympy string, use it as Python code:
def myfunction(x, A1, xc1, sigma1):
return A1*exp(-(x-xc1)**2/(2*sigma1**2))
and then
from lmfit import Model
mymodel = Model(myfunction)
params = mymodel.guess(A1=2, xc1=2, sigma1=1)
In short: sympy is an amazing tool, but lmfit does not use it.
When I run this code
import Scientific.IO.NetCDF as S
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import matplotlib.pyplot as plt
import xarray as xr
import metpy
import numpy as N
from metpy.plots import ContourPlot, ImagePlot, MapPanel, PanelContainer
# Any import of metpy will activate the accessors
import metpy.calc as mpcalc
#from metpy.testing import get_test_data
from metpy.units import units
# Open the netCDF file as a xarray Datase
#
datadir='C:/Users/stratus/AppData/Local/lxss/home/stratus/PROJECT/NEWPROJECT/FEB012017/nam_218_20170131_1200_000.nc'
data = xr.open_dataset(datadir,decode_cf=True)
# To parse the full dataset, we can call parse_cf without an argument, and assign the returned
# Dataset.
data = data.metpy.parse_cf()
tempatt=data['TMP_P0_L100_GLC0'].attrs
# If we instead want just a single variable, we can pass that variable name to parse_cf and
# it will return just that data variable as a DataArray.
data_var = data.metpy.parse_cf('TMP_P0_L100_GLC0')
# To rename variables, supply a dictionary between old and new names to the rename method
data.rename({
'TMP_P0_L100_GLC0': 'temperature',
}, inplace=True)
data['temperature'].metpy.convert_units('degC')
# Get multiple coordinates (for example, in just the x and y direction)
x, y = data['temperature'].metpy.coordinates('x', 'y')
# If we want to get just a single coordinate from the coordinates method, we have to use
# tuple unpacking because the coordinates method returns a generator
vertical, = data['temperature'].metpy.coordinates('vertical')
data_crs = data['temperature'].metpy.cartopy_crs
# Or, we can just get a coordinate from the property
#time = data['temperature'].metpy.time
# To verify, we can inspect all their names
#print([coord.name for coord in (x, y, vertical, time)])
#
#heights = data['height'].metpy.loc[{'time': time[0], 'vertical': 850. * units.hPa}]
#lat, lon = xr.broadcast(y, x)
#f = mpcalc.coriolis_parameter(lat)
#dx, dy = mpcalc.grid_deltas_from_dataarray(heights)
#u_geo, v_geo = mpcalc.geostrophic_wind(heights, f, dx, dy)
#print(u_geo)
#print(v_geo)
fig=plt.figure(1)
# A very simple example example of a plot of 500 hPa heights
data_crs = data['temperature'].metpy.cartopy_crs
ax = plt.axes(projection=ccrs.LambertConformal())
data['temperature'].metpy.loc[{'vertical': 850. * units.hPa}].plot(ax=ax, transform=data_crs)
ax.add_feature(cfeature.LAND)
ax.add_feature(cfeature.OCEAN)
ax.add_feature(cfeature.COASTLINE)
plt.show()
#ax.set_extent([-120,-80,20,50])
plt.title("850 mb Temperature")
#plt.suptitle("Metpy Test")
plt.show()
I had to edit the code as per some of the answers but I am getting a mostly blank map now. 850 T Map fail I am mainly trying to have the temperatures at 850 mb overlap the US so I could show it to a friend to practice for a project I am helping him with. The filling of the parentheses for the data helped a bit which is why I edited it.
As pointed out in the comments it is difficult to answer without a reproducible example. However, the following may solve your issue:
data_crs = data['temperature'].metpy.cartopy_crs
ax = plt.axes(projection=ccrs.LambertConformal())
data['temperature'].metpy.loc[{'vertical': 1000. * units.hPa}].plot(ax=ax, transform=data_crs)
ax.add_feature(cfeature.LAND)
ax.add_feature(cfeature.OCEAN)
ax.add_feature(cfeature.COASTLINE)
plt.show()
I have seen many examples of integrating matplotlib with python2.6 and QT4 Designer using mplwidget and they work fine. I now need to integrate a pyplot with QT4 Designer but cannot find any examples. All of my attempts to integrate a pyplot graphic have ended in a seg fault. Can someone please direct me to a working example using Designer and pyplot?
Follow up:
Okay, so I tried your solution but I'm still having issues. Unfortunately the machine I use for this code is not hooked up to the internet, so below is a fat finger of the pertinent parts of the code I am using:
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import MatplotlibWidget # This is the code snippet you gave in your answer
.
.
.
def drawMap(self) :
fig = plt.figure()
map = Basemap(projection='cyl', llcrnrlat = -90.0, urcrnrlat = 90.0, llcrnrlon = -180.0, urcrnrlon = 180.0, resolution = 'c')
map.drawcoastlines()
map.drawcountries()
plt.show()
def drawMap_Qt(self) :
self.ui.PlotWidget.figure = plt.figure() # This is the Qt widget promoted to MatplotlibWidget
map = Basemap(projection='cyl', llcrnrlat = -90.0, urcrnrlat = 90.0, llcrnrlon = -180.0, urcrnrlon = 180.0, resolution = 'c')
map.drawcoastlines()
map.drawcountries()
self.ui.PlotWidget.show()
The function drawMap() works fine. It creates a separate window and plots the map. The drawMap_Qt() function results in a segmentation fault with no other errors. The end goal is to plot a contour on top of the map. I can do this with the drawMap() function. Of course, I can't even get to the contour part with the drawMap_Qt() function. Any insights as to why it is seg faulting would be greatly appreciated.
If you're referring to the mplwidget from Python(x,y) and WinPython, I think it does use pyplot, but I had trouble putting it in my python install so I just used this class:
from PyQt4 import QtGui
from matplotlib.figure import Figure
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
from matplotlib.backends.backend_qt4agg import NavigationToolbar2QT as NavigationToolbar
class MatplotlibWidget(QtGui.QWidget):
def __init__(self, parent=None, *args, **kwargs):
super(MatplotlibWidget, self).__init__(parent)
self.figure = Figure(*args, **kwargs)
self.canvas = FigureCanvas(self.figure)
self.toolbar = NavigationToolbar(self.canvas, self)
layout = QtGui.QVBoxLayout()
layout.addWidget(self.toolbar)
layout.addWidget(self.canvas)
self.setLayout(layout)
See also How to embed matplotib in pyqt - for Dummies.
Then in Qt Designer you need to create a promoted class, with base class: QWidget, promoted class name: MatplotlibWidget, and header file: the python script containing the MatplotlibWidget class (without the .py). You can add things like ax = self.figure.add_subplot(111), line = ax.plt(...) within the class or by calling methods of the figure attribute of an instance of the class.
Edit:
So I was a bit wrong before, in general with embedded matplotlib widgets you need to use the object oriented methods and not the functions in pyplot. (This sort of explains what the difference is.) Using my snippet above as mymatplotlibwidget.py, try something like this. I don't have basemap installed, so this is somewhat of a guess, but from the examples you need to tell Basemap which axes to use.
import sys
from PyQt4 import QtGui
from mpl_toolkits.basemap import Basemap
from mymatplotlibwidget import MatplotlibWidget
app = QtGui.QApplication(sys.argv)
widget = MatplotlibWidget()
fig = widget.figure
ax = fig.add_subplot(111)
map = Basemap(..., ax=ax)
fig.canvas.draw()
widget.show()
app.exec_()
I'm making a big dendrogram using SciPy and in the resulting dendrogram the line thickness makes it hard to see detail. I want to decrease the line thickness to make it easier to see and more MatLab like. Any suggestions?
I'm doing:
import scipy.cluster.hierarchy as hicl
from pylab import savefig
distance = #distance matrix
links = hicl.linkage(distance,method='average')
pden = hicl.dendrogram(links,color_threshold=optcutoff[0], ...
count_sort=True,no_labels=True)
savefig('foo.pdf')
And getting a result like this.
Matplotlib has a context manager now, which allows you to only override the default values temporarily, for that one plot:
import matplotlib.pyplot as plt
from scipy.cluster import hierarchy
distance = #distance matrix
links = hierarchy.linkage(distance, method='average')
# Temporarily override the default line width:
with plt.rc_context({'lines.linewidth': 0.5}):
pden = hierarchy.dendrogram(links, color_threshold=optcutoff[0], ...
count_sort=True, no_labels=True)
# linewidth is back to its default here...!
plt.savefig('foo.pdf')
See the Matplotlib configuration API for more details.
Set the default linewidth before calling dendrogram. For example:
import scipy.cluster.hierarchy as hicl
from pylab import savefig
import matplotlib
# Override the default linewidth.
matplotlib.rcParams['lines.linewidth'] = 0.5
distance = #distance matrix
links = hicl.linkage(distance,method='average')
pden = hicl.dendrogram(links,color_threshold=optcutoff[0], ...
count_sort=True,no_labels=True)
savefig('foo.pdf')
See Customizing matplotlib for more information.
set dendrogram on existing axes than change its artists using setp. It allow changing all parameters, that won't work if dendrogram is sent to axes or won't work with dendrogram at all like linestyle.
import matplotlib.pyplot as plt
import scipy.cluster.hierarchy as hicl
links = #linkage
fig,ax = plt.subplots()
hicl.dendrogram(links,ax=ax)
plt.setp(ax.collections,linewidth=3,linestyle=":", ...other line parameters...)