I got the following error:
<lambdifygenerated-1>:2: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.return numpy.array((A1exp(-1/2(x - xc1)**2/sigma1**2), 0, 0))
Here I have just one model but this code is written for model combination in fitting by the lmfit Please kindly let me know about it.
import matplotlib.pyplot as plt
import numpy as np
import sympy
from sympy.parsing import sympy_parser
import lmfit
gauss_peak1 = sympy_parser.parse_expr('A1*exp(-(x-xc1)**2/(2*sigma1**2))')
gauss_peak2 = 0
exp_back = 0
model_list = sympy.Array((gauss_peak1, gauss_peak2, exp_back))
model = sum(model_list)
print(model)
model_list_func = sympy.lambdify(list(model_list.free_symbols), model_list)
model_func = sympy.lambdify(list(model.free_symbols), model)
np.random.seed(1)
x = np.linspace(0, 10, 40)
param_values = dict(x=x, A1=2, sigma1=1, xc1=2)
y = model_func(**param_values)
yi = model_list_func(**param_values)
yn = y + np.random.randn(y.size)*0.4
plt.plot(x, yn, 'o')
plt.plot(x, y)
lm_mod = lmfit.Model(model_func, independent_vars=('x'))
res = lm_mod.fit(data=yn, **param_values)
res.plot_fit()
plt.plot(x, y, label='true')
plt.legend()
plt.show()
lmfit.Model takes a model function that is a Python function. It parses the function arguments and expects those to be the Parameters for the model.
I don't think using sympy-created functions will do that. Do you need to use sympy here? I don't see why. The usage here seems designed to make the code more complex than it needs to be. It seems you want to make a model with a Gaussian-like peak, and a constant(?) background. If so, why not do
from lmfit.Models import GaussianModel, ConstantModel
model = GaussianModel(prefix='p1_') + ConstantModel()
params = model.make_params(p1_amplitude=2, p1_center=2, p1_sigma=1, c=0)
That just seems way easier to me, and it is very easy to add a second Gaussian peak to that model.
But even if you have your own preferred mathematical expression, don't use that as a sympy string, use it as Python code:
def myfunction(x, A1, xc1, sigma1):
return A1*exp(-(x-xc1)**2/(2*sigma1**2))
and then
from lmfit import Model
mymodel = Model(myfunction)
params = mymodel.guess(A1=2, xc1=2, sigma1=1)
In short: sympy is an amazing tool, but lmfit does not use it.
Related
I have a dataset which I have to process in such a way that it works with a convolutional neural network of PyTorch (I'm completely new to PyTorch). The data is stored in a dataframe with a column for pictures (28 x 28 ndarrays with int32 entries) and a column with its class labels. The pixels of the images merely adopt values +1 and -1 (since it is simulation data of a classical 2d Ising Model). The dataframe looks like this.
I imported the following (a lot of this is not relevant for now, but I included everything for completeness. "data_loader" is a custom py file.):
import numpy as np
import matplotlib.pyplot as plt
import data_loader
import pandas as pd
import torch
import torchvision.transforms as T
from torchvision.utils import make_grid
from torch.nn import Module
from torch.nn import Conv2d
from torch.nn import Linear
from torch.nn import MaxPool2d
from torch.nn import ReLU
from torch.nn import LogSoftmax
from torch import flatten
from sklearn.metrics import classification_report
import time as time
from torch.utils.data import DataLoader, Dataset
Then, I want to get this in the correct shape in order to make it useful for PyTorch. I do this by defining the following class
class MetropolisDataset(Dataset):
def __init__(self, data_frame, transform=None):
self.data_frame = data_frame
self.transform = transform
def __len__(self):
return len(self.data_frame)
def __getitem__(self,idx):
if torch.is_tensor(idx):
idx = idx.tolist()
label = self.data_frame['label'].iloc[idx]
image = self.data_frame['image'].iloc[idx]
image = np.array(image)
if self.transform:
image = self.transform(image)
return (image, label)
I call instances of this class as:
train_set = MetropolisDataset(data_frame = df_train,
transform = T.Compose([
T.ToPILImage(),
T.ToTensor()]))
validation_set = MetropolisDataset(data_frame = df_validation,
transform = T.Compose([
T.ToPILImage(),
T.ToTensor()]))
test_set = MetropolisDataset(data_frame = df_test,
transform = T.Compose([
T.ToPILImage(),
T.ToTensor()]))
The problem does not yet arise here, because I am able to read out and show images from these instances of the above defined class.
Then, as far as I found out, it is necessary to let this go through the DataLoader of PyTorch, which I do as follows:
batch_size = 64
train_dl = DataLoader(train_set, batch_size, shuffle=True, num_workers=3, pin_memory=True)
validation_dl = DataLoader(validation_set, batch_size, shuffle=True, num_workers=3, pin_memory=True)
test_dl = DataLoader(test_set, batch_size, shuffle=True, num_workers=3, pin_memory=True)
However, if I want to use these instances of the DataLoader, simply nothing happens. I neither get an error, nor the computation seems to get anywhere. I tried to run a CNN but it does not seem to compute anything. Something else I tried was to show some sample images with the code provided by this article, but the same issue occurs. The sample code is:
def show_images(images, nmax=10):
fig, ax = plt.subplots(figsize=(8, 8))
ax.set_xticks([]); ax.set_yticks([])
ax.imshow(make_grid((images.detach()[:nmax]), nrow=8).permute(1, 2, 0))
def show_batch(dl, nmax=64):
for images in dl:
show_images(images, nmax)
break
show_batch(test_dl)
It seems that there is some error in the implementation of my MetropolisDataset class or with the DataLoader itself. How could this problem be solved?
As mentioned in the comments, the problem was partly solved by setting num_workers to zero since I was working in a Jupyter notebook, as answered here. However, this left open one further problem that I got errors when I wanted to apply the DataLoader to run a CNN. The issue was then that my data did consist of int32 numbers instead of float32. I do not include further codes, because this was related directly to my data - however, the issue was (as very often) merely a wrong datatype.
I'm working on fitting of the experimental data. In order to fit it I use the minimization of the function of residual. Everything is quite trivial, but but this time I can't find what's wrong and why the result of fitting is so weird. The example is simplified in comparison with original problem. But anyway it gives wrong parameters even when I set used values of parameters as initial guess.
import matplotlib.pyplot as plt
import numpy as np
import csv
from scipy.optimize import curve_fit, minimize
x=np.arange(0,10,0.5)
a=0.5
b=3
ini_pars=[a, b]
def func(x, a, b):
return a*x+b
plt.plot(x, func(x,a,b))
plt.show()
def fit(pars):
A,B = pars
res = (func(x,a, b)-func(x, *pars))**2
s=sum(res)
return s
bnds=[(0.1,0.5),(1,5)]
x0=[0.1,4]
opt = minimize(fit, x0, bounds=bnds)
new_pars=[opt.x[0], opt.x[0]]
example = fit(ini_pars)
print(example)
example = fit(new_pars)
print(example)
print(new_pars)
plt.plot(x, func(x, *ini_pars))
plt.plot(x, func(x, *new_pars))
plt.show()
```[enter image description here][1]
[1]: https://i.stack.imgur.com/qc1Nu.png
It should be new_pars=[opt.x[0], opt.x[1]] instead of new_pars=[opt.x[0], opt.x[0]]. Note also that you can directly extract the values by new_pars = opt.x.
I am trying to get a GLCM implementation running in a custom Keras Layer in a reasonable fast time. So far I took the _glcm_loop from skimage-implementation, reduced it to what I needed and put it into a basic layer, like this:
import numpy as np
import tensorflow as tf
from time import time
from tensorflow import keras
from tensorflow.keras.preprocessing import image
from tensorflow.keras import layers
from skimage.feature import *
from numpy import array
from math import sin, cos
from time import time
import matplotlib.pyplot as plt
class GLCMLayer(keras.layers.Layer):
def __init__(self, greylevels=32, angles=[0], distances=[1], name=None, **kwargs):
self.greylevels = greylevels
self.angles = angles
self.distances = distances
super(GLCMLayer, self).__init__(name=name, **kwargs)
def _glcm_loop(self, image, distances, angles, levels, out):
rows = image.shape[0]
cols = image.shape[1]
for a_idx in range(len(angles)):
angle = angles[a_idx]
for d_idx in range(len(distances)):
distance = distances[d_idx]
offset_row = round(sin(angle) * distance)
offset_col = round(cos(angle) * distance)
start_row = max(0, -offset_row)
end_row = min(rows, rows - offset_row)
start_col = max(0, -offset_col)
end_col = min(cols, cols - offset_col)
for r in range(start_row, end_row):
for c in range(start_col, end_col):
i = image[r, c]
row = r + offset_row
col = c + offset_col
j = image[row, col]
out[i, j, d_idx, a_idx] += 1
def call(self, inputs):
P = np.zeros((self.greylevels, self.greylevels, len(self.distances), len(self.angles)), dtype=np.uint32, order='C')
self._glcm_loop(inputs, self.distances, self.angles, self.greylevels, P)
return P
def get_config(self):
config = {
'angle': self.angle,
'distance': self.distance,
'greylevel': self.greylevel,
}
base_config = super(GLCMLayer, self).get_config()
return dict(list(base_config.items()) + list(config.items()))
My execution code looks like this:
def quant(img, greylevels):
return array(img)//(256//greylevels)
if __name__ == "__main__":
source_file = "<some sour file>"
img_raw = image.load_img(source_file, target_size=(150,150), color_mode="grayscale")
img = quant(img_raw, 32)
layer = GLCMLayer()
start = time()
aug = layer(img)
tf.print(time()-start)
This is my first step to create it as a preprocessing layer. The second step then will be to modify it to run it also as hidden layer inside a model. That is why I didn't put it to a complete model yet, but I feel like there will be additional changes required when doing so.
For some reason the execution time is about 15-20 seconds long. Executing the code on the CPU without the layer takes about 0.0009 seconds. Obviously, something is going wrong here.
I am fairly new to tf and keras, so I fear I am missing something in the way on how to use the framework. In order to resolve it, I read about (which doesn't mean I understood):
do not use np-functions inside tensorflow, but tf-functions instead,
use tf.Variable,
use tf.Data,
unfolding is not possible in some way (whatever that means)
I tried a little here and there, but couldn't get them running, instead finding various different exceptions. So my questions are:
What is the correct way to use tf-functions in a GLCM to get the best performance on the GPU?
What do I need to take care of when using the layer in a complete model?
From that point on, I should hopefully be able to then implement the GLCM properties.
Any help is greatly appreciated.
(Disclaimer: I assume that there is a lot of other stuff not optimal yet, if anything comes to your mind just add it.)
I am trying to implement the 'Iterative hessian Sketch' algorithm from https://arxiv.org/abs/1411.0347 page 12. However, I am struggling with step two which needs to minimize the matrix-vector function.
Imports and basic data generating function
import numpy as np
import scipy as sp
from sklearn.datasets import make_regression
from scipy.optimize import minimize
import matplotlib.pyplot as plt
%matplotlib inline
from numpy.linalg import norm
def generate_data(nsamples, nfeatures, variance=1):
'''Generates a data matrix of size (nsamples, nfeatures)
which defines a linear relationship on the variables.'''
X, y = make_regression(n_samples=nsamples, n_features=nfeatures,\
n_informative=nfeatures,noise=variance)
X[:,0] = np.ones(shape=(nsamples)) # add bias terms
return X, y
To minimize the matrix-vector function, I have tried implementing a function which computes the quanity I would like to minimise:
def f2min(x, data, target, offset):
A = data
S = np.eye(A.shape[0])
#S = gaussian_sketch(nrows=A.shape[0]//2, ncols=A.shape[0] )
y = target
xt = np.ravel(offset)
norm_val = (1/2*S.shape[0])*norm(S#A#(x-xt))**2
#inner_prod = (y - A#xt).T#A#x
return norm_val - inner_prod
I would eventually like to replace S with some random matrices which can reduce the dimensionality of the problem, however, first I need to be confident that this optimisation method is working.
def grad_f2min(x, data, target, offset):
A = data
y = target
S = np.eye(A.shape[0])
xt = np.ravel(offset)
S_A = S#A
grad = (1/S.shape[0])*S_A.T#S_A#(x-xt) - A.T#(y-A#xt)
return grad
x0 = np.zeros((X.shape[0],1))
xt = np.zeros((2,1))
x_new = np.zeros((2,1))
for it in range(1):
result = minimize(f2min, x0=xt,args=(X,y,x_new),
method='CG', jac=False )
print(result)
x_new = result.x
I don't think that this loop is correct at all because at the very least there should be some local convergence before moving on to the next step. The output is:
fun: 0.0
jac: array([ 0.00745058, 0.00774882])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 416
nit: 0
njev: 101
status: 2
success: False
x: array([ 0., 0.])
Does anyone have an idea if:
(1) Why I'm not achieving convergence at each step
(2) I can implement step 2 in a better way?
You may already know, that in matplotlib 1.2.0 there is a new experimental feature, that figures are pickable (they can be saved with pickle module).
However, it doesn't work when one uses logscale, eg.
import matplotlib.pyplot as plt
import numpy as np
import pickle
ax = plt.subplot(111)
x = np.linspace(0, 10)
y = np.exp(x)
plt.plot(x, y)
ax.set_yscale('log')
pickle.dump(ax, file('myplot.pickle', 'w'))
results in:
PicklingError: Can't pickle <class 'matplotlib.scale.Log10Transform'>: attribute lookup matplotlib.scale.Log10Transform failed
Anybody knows any solution/workaround to this?
I've opened this as a bug report on matplotlib's github issue tracker. Its a fairly easy fix to implement on the matplotlib repository side (simply don't nest the Log10Transform class inside the LogScale class), but that doesn't really help you in being able to use this with mpl 1.2.0...
There is a solution to getting this to work for you in 1.2.0, but I warn you - its not pretty!
Based on my answer to a pickling question it is possible to pickle nested classes (as Log10Transform is). All we need to do is to tell Log10Transform how to "reduce" itself:
import matplotlib.scale
class _NestedClassGetter(object):
"""
When called with the containing class as the first argument,
the name of the nested class as the second argument,
and the state of the object as the third argument,
returns an instance of the nested class.
"""
def __call__(self, containing_class, class_name, state):
nested_class = getattr(containing_class, class_name)
# return an instance of a nested_class. Some more intelligence could be
# applied for class construction if necessary.
c = nested_class.__new__(nested_class)
c.__setstate__(state)
return c
def _reduce(self):
# return a class which can return this class when called with the
# appropriate tuple of arguments
cls_name = matplotlib.scale.LogScale.Log10Transform.__name__
call_args = (matplotlib.scale.LogScale, cls_name, self.__getstate__())
return (_NestedClassGetter(), call_args)
matplotlib.scale.LogScale.Log10Transform.__reduce__ = _reduce
You might also decide to do this for other Log based transforms/classes, but for your example, you can now pickle (and successfully unpickle) your example figure:
import matplotlib.pyplot as plt
import numpy as np
import pickle
ax = plt.subplot(111)
x = np.linspace(0, 10)
y = np.exp(x)
plt.plot(x, y)
ax.set_yscale('log')
pickle.dump(ax, file('myplot.pickle', 'w'))
plt.savefig('pickle_log.pre.png')
plt.close()
pickle.load(file('myplot.pickle', 'r'))
plt.savefig('pickle_log.post.png')
I'm going to get on and fix this for mpl 1.3.x so that this nasty workaround isn't needed in the future :-) .
HTH,