output[0] = y0 ValueError: setting an array element with a sequence - numpy

I have been struggling for a few days with this. I'm trying t estimate the density of a piecewise gaussian function. Could anyone tell me why I'm now getting the error
TypeError: output[0] = y0
ValueError: setting an array element with a sequence.
It happens on this line:
Zero_RG = integrate.romberg(gaussian(q,x,mu,sigma), Q1, Q2).`
Here is the script:
import numpy as np
import sympy as sp
from sympy import *
from scipy import integrate
from sympy import Integral, log, exp, sqrt, pi
import matplotlib.pyplot as plt
from scipy.stats import norm, gaussian_kde
from quantecon import LAE
from sympy import symbols
var('Q1 Q2 x q sigma mu')
#q= symbols('q')
## == Define parameters == #
mu=80
sigma=20
b=0.2
Q=80
Q1=Q*(1-b)
Q2=Q*(1+b)
d = (sigma*np.sqrt(2*np.pi))
phi = norm()
n = 500
def p(x, y):
x, y = np.array(x, dtype=float), np.array(y, dtype=float)
Positive_RG = norm.pdf(x-y+Q1, mu, sigma)
print('Positive_R = ', Positive_RG)
Negative_RG = norm.pdf(x-y+Q2, mu, sigma)
print('Negative_RG = ', Negative_RG)
gaussian = lambda q,x,mu,sigma: 1/(sigma*np.sqrt(2*np.pi))*np.exp(-(x+q-mu)**2 /(2*sigma**2))
wrapped_gaussian = lambda q: gaussian(q, x, mu, sigma)
Zero_RG = integrate.romberg(wrapped_gaussian, Q1, Q2)
print('pdf',gaussian)
#Zero_RG = scipy.integrate.quad(norm.pdf(x + q, mu, sigma))
# Int_zerocase= lambda q: norm.pdf(x + q, u, sigma)
# Zero_RG = scipy.integrate.quad(Int_zerocase, Q1, Q2)
# print(Zero_RG)
if y>0.0 and x -y>=-Q1:
#print('printA', Positive_RG)
return Positive_RG
elif y<0.0 and x -y>=-Q2:
#print('printC', Negative_RG)
return Negative_RG
elif y==0.0 and x >=-Q1:
#print('printB', Zero_RG)
return Zero_RG
return 0.0
Z = phi.rvs(n)
X = np.empty(n)
for t in range(n-1):
X[t+1] = X[t] + Z[t]
#X[t+1] = np.abs(X[t]) + Z[t]
psi_est = LAE(p, X)
k_est = gaussian_kde(X)
fig, ax = plt.subplots(figsize=(10,7))
ys = np.linspace(-200.0, 200.0, 200)
ax.plot(ys, psi_est(ys), 'g-', lw=2, alpha=0.6, label='look ahead estimate')
ax.plot(ys, k_est(ys), 'k-', lw=2, alpha=0.6, label='kernel based estimate')
ax.legend(loc='upper left')
plt.show()

The docs for romberg are pretty clear that the first argument is a function of a single variable that gets integrated.
First, a minor point. Use np.exp in preference to np.e**.
In Python, the expression
gaussian = lambda q,x,mu,sigma: 1/(sigma*np.sqrt(2*np.pi))*np.exp(-(x+q-mu)**2 /(2*sigma**2))
sets gaussian to a function of four arguments. The expression gaussian(q, x, mu, sigma) is just the return value of that function.
There are two ways to pass in the required parameters to the romberg. The easiest way is to use the args parameter to pass in the three additional arguments as a tuple:
Zero_RG = integrate.romberg(gaussian, Q1, Q2, args=(x,mu,sigma))
The other way is to create a wrapper function that will pass the arguments for you:
wrapped_gaussian = lambda q: gaussian(q, x, mu, sigma)
Zero_RG = integrate.romberg(wrapped_gaussian, Q1, Q2)
I would recommend the fist approach because it uses an existing mechanism, as well as being shorter and easier to read.

Related

How do I vectorize a function in numpy with some fixed parameters?

I have written a code for approximating a function with the Bernstein polynomials ( https://en.wikipedia.org/wiki/Bernstein_polynomial )
at
https://github.com/pdenapo/metodos-numericos/blob/master/python/bernstein.py
I have a function that gives the polynomial approximating f as bernstein(f, n, p) (where f is the function that I want to approximate, n is the degree and p the point where it is evaluated.
def bernstein(f, n, p):
return np.sum(
[f(k / n) * st.binom.pmf(k, n, p) for k in np.arange(0, n + 1)])
Now I want to generate a plot of this function where f and n es fixed, and p runs though a vector generated by np.arrange
So I am vectorizing the function in the following way:
bernstein3 = lambda x: bernstein(f, 3, x)
bernstein3 = np.vectorize(bernstein3)
y3 = bernstein3(x)
plt.plot(x, y3, 'green', label='$B_3$')
It works. But I guess there must be some more elegant, or perhaps more pythonic way of doing this. Any suggestions? Many thanks
Since SciPy statistic functions are vectorized, your bernstein function can be modified in a straightforward manner to work that way:
import numpy as np
import scipy.stats
def bernstein(f, n, p):
# Vector of k values
k = np.arange(n + 1)
# Add a broadcasting dimension to p
pd = np.expand_dims(p, -1)
# Compute approximation
return np.sum(f(k / n) * scipy.stats.binom.pmf(k, n, pd), -1)
It would be used simply as this:
import numpy as np
import matplotlib.pyplot as plt
def f(x):
return np.abs(1 / 2 - x)
x = np.linspace(0, 1, 100)
y = f(x)
plt.plot(x, y, 'blue', label='f(x)')
y_approx = bernstein(f, 10, x)
plt.plot(x, y_approx, 'orange', label='f_approx(x)')
plt.show()

Implementing minimization in SciPy

I am trying to implement the 'Iterative hessian Sketch' algorithm from https://arxiv.org/abs/1411.0347 page 12. However, I am struggling with step two which needs to minimize the matrix-vector function.
Imports and basic data generating function
import numpy as np
import scipy as sp
from sklearn.datasets import make_regression
from scipy.optimize import minimize
import matplotlib.pyplot as plt
%matplotlib inline
from numpy.linalg import norm
def generate_data(nsamples, nfeatures, variance=1):
'''Generates a data matrix of size (nsamples, nfeatures)
which defines a linear relationship on the variables.'''
X, y = make_regression(n_samples=nsamples, n_features=nfeatures,\
n_informative=nfeatures,noise=variance)
X[:,0] = np.ones(shape=(nsamples)) # add bias terms
return X, y
To minimize the matrix-vector function, I have tried implementing a function which computes the quanity I would like to minimise:
def f2min(x, data, target, offset):
A = data
S = np.eye(A.shape[0])
#S = gaussian_sketch(nrows=A.shape[0]//2, ncols=A.shape[0] )
y = target
xt = np.ravel(offset)
norm_val = (1/2*S.shape[0])*norm(S#A#(x-xt))**2
#inner_prod = (y - A#xt).T#A#x
return norm_val - inner_prod
I would eventually like to replace S with some random matrices which can reduce the dimensionality of the problem, however, first I need to be confident that this optimisation method is working.
def grad_f2min(x, data, target, offset):
A = data
y = target
S = np.eye(A.shape[0])
xt = np.ravel(offset)
S_A = S#A
grad = (1/S.shape[0])*S_A.T#S_A#(x-xt) - A.T#(y-A#xt)
return grad
x0 = np.zeros((X.shape[0],1))
xt = np.zeros((2,1))
x_new = np.zeros((2,1))
for it in range(1):
result = minimize(f2min, x0=xt,args=(X,y,x_new),
method='CG', jac=False )
print(result)
x_new = result.x
I don't think that this loop is correct at all because at the very least there should be some local convergence before moving on to the next step. The output is:
fun: 0.0
jac: array([ 0.00745058, 0.00774882])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 416
nit: 0
njev: 101
status: 2
success: False
x: array([ 0., 0.])
Does anyone have an idea if:
(1) Why I'm not achieving convergence at each step
(2) I can implement step 2 in a better way?

getting matplotlib radar plot with pandas

I am trying to go a step further by creating a radar plot like this question states. I using the same source code that the previous question was using, except I'm trying to implement this using pandas dataframe and pivot tables.
import numpy as np
import pandas as pd
from StringIO import StringIO
import matplotlib.pyplot as plt
from matplotlib.projections.polar import PolarAxes
from matplotlib.projections import register_projection
def radar_factory(num_vars, frame='circle'):
"""Create a radar chart with `num_vars` axes."""
# calculate evenly-spaced axis angles
theta = 2 * np.pi * np.linspace(0, 1 - 1. / num_vars, num_vars)
# rotate theta such that the first axis is at the top
theta += np.pi / 2
def draw_poly_frame(self, x0, y0, r):
# TODO: use transforms to convert (x, y) to (r, theta)
verts = [(r * np.cos(t) + x0, r * np.sin(t) + y0) for t in theta]
return plt.Polygon(verts, closed=True, edgecolor='k')
def draw_circle_frame(self, x0, y0, r):
return plt.Circle((x0, y0), r)
frame_dict = {'polygon': draw_poly_frame, 'circle': draw_circle_frame}
if frame not in frame_dict:
raise ValueError, 'unknown value for `frame`: %s' % frame
class RadarAxes(PolarAxes):
"""Class for creating a radar chart (a.k.a. a spider or star chart)
http://en.wikipedia.org/wiki/Radar_chart
"""
name = 'radar'
# use 1 line segment to connect specified points
RESOLUTION = 1
# define draw_frame method
draw_frame = frame_dict[frame]
def fill(self, *args, **kwargs):
"""Override fill so that line is closed by default"""
closed = kwargs.pop('closed', True)
return super(RadarAxes, self).fill(closed=closed, *args, **kwargs)
def plot(self, *args, **kwargs):
"""Override plot so that line is closed by default"""
lines = super(RadarAxes, self).plot(*args, **kwargs)
for line in lines:
self._close_line(line)
def _close_line(self, line):
x, y = line.get_data()
# FIXME: markers at x[0], y[0] get doubled-up
if x[0] != x[-1]:
x = np.concatenate((x, [x[0]]))
y = np.concatenate((y, [y[0]]))
line.set_data(x, y)
def set_varlabels(self, labels):
self.set_thetagrids(theta * 180 / np.pi, labels)
def _gen_axes_patch(self):
x0, y0 = (0.5, 0.5)
r = 0.5
return self.draw_frame(x0, y0, r)
register_projection(RadarAxes)
return theta
def day_radar_plot(df):
fig = plt.figure(figsize=(6,6))
#adjust spacing around the subplots
fig.subplots_adjust(wspace=0.25,hspace=0.20,top=0.85,bottom=0.05)
ldo,rup = 0.1,0.8 #leftdown and right up normalized
ax = fig.add_axes([ldo,ldo,rup,rup],polar=True)
N = len(df['Group1'].unique())
theta = radar_factory(N)
polar_df = pd.DataFrame(df.groupby([df['Group1'],df['Type'],df['Vote']]).size())
polar_df.columns = ['Count']
radii = polar_df['Count'].get_values()
names = polar_df.index.get_values()
#get the number of unique colors needed
num_colors_needed = len(names)
#Create the list of unique colors needed for red and blue shades
Rcolors = []
Gcolors = []
for i in range(num_colors_needed):
ri=1-(float(i)/float(num_colors_needed))
gi=0.
bi=0.
Rcolors.append((ri,gi,bi))
for i in range(num_colors_needed):
ri=0.
gi=1-(float(i)/float(num_colors_needed))
bi=0.
Gcolors.append((ri,gi,bi))
from_x = np.linspace(0,0.95,num_colors_needed)
to_x = from_x + 0.05
i = 0
for d,f,R,G in zip(radii,polar_df.index,Rcolors,Gcolors):
i = i+1
if f[2].lower() == 'no':
ax.plot(theta,d,color=R)
ax.fill(theta,d,facecolor=R,alpha=0.25)
#this is where I think i have the issue
ax.axvspan(from_x[i],to_x[i],color=R)
elif f[2].lower() == 'yes':
ax.plot(theta,d,color=G)
ax.fill(theta,d,facecolor=G,alpha=0.25)
#this is where I think i have the issue
ax.axvspan(from_x[i],to_x[i],color=G)
plt.show()
So, let's say I have this StringIO that has a list of Group1 voting either yes or no and they are from a numbered type..these numbers are arbitrary in labeling but just as an example..
fakefile = StringIO("""\
Group1,Type,Vote
James,7,YES\nRachael,7,YES\nChris,2,YES\nRachael,9,NO
Chris,2,YES\nChris,7,NO\nRachael,9,NO\nJames,2,NO
James,7,NO\nJames,9,YES\nRachael,9,NO
Chris,2,YES\nChris,2,YES\nRachael,7,NO
Rachael,7,YES\nJames,9,YES\nJames,9,NO
Rachael,2,NO\nChris,2,YES\nRachael,7,YES
Rachael,9,NO\nChris,9,NO\nJames,7,NO
James,2,YES\nChris,2,NO\nRachael,9,YES
Rachael,9,YES\nRachael,2,NO\nChris,7,YES
James,7,YES\nChris,9,NO\nRachael,9,NO\n
Chris,9,YES
""")
record = pd.read_csv(fakefile, header=0)
day_radar_plot(record)
The error I get is Value Error: x and y must have same first dimension.
As I indicated in my script, I thought I had a solution for it but apparently I'm going by it the wrong way. Does anyone have any advice or guidance?
Since I'm completely lost in what you are trying to do, I will simply provide a solution on how to draw a radar chart from the given data.
It will answer the question how often have people voted Yes or No.
import pandas as pd
import numpy as np
from StringIO import StringIO
import matplotlib.pyplot as plt
fakefile = StringIO("""\
Group1,Type,Vote
James,7,YES\nRachael,7,YES\nChris,2,YES\nRachael,9,NO
Chris,2,YES\nChris,7,NO\nRachael,9,NO\nJames,2,NO
James,7,NO\nJames,9,YES\nRachael,9,NO
Chris,2,YES\nChris,2,YES\nRachael,7,NO
Rachael,7,YES\nJames,9,YES\nJames,9,NO
Rachael,2,NO\nChris,2,YES\nRachael,7,YES
Rachael,9,NO\nChris,9,NO\nJames,7,NO
James,2,YES\nChris,2,NO\nRachael,9,YES
Rachael,9,YES\nRachael,2,NO\nChris,7,YES
James,7,YES\nChris,9,NO\nRachael,9,NO\n
Chris,9,YES""")
df = pd.read_csv(fakefile, header=0)
df["cnt"] = np.ones(len(df))
pt = pd.pivot_table(df, values='cnt', index=['Group1'],
columns=['Vote'], aggfunc=np.sum)
fig = plt.figure()
ax = fig.add_subplot(111, projection="polar")
theta = np.arange(len(pt))/float(len(pt))*2.*np.pi
l1, = ax.plot(theta, pt["YES"], color="C2", marker="o", label="YES")
l2, = ax.plot(theta, pt["NO"], color="C3", marker="o", label="NO")
def _closeline(line):
x, y = line.get_data()
x = np.concatenate((x, [x[0]]))
y = np.concatenate((y, [y[0]]))
line.set_data(x, y)
[_closeline(l) for l in [l1,l2]]
ax.set_xticks(theta)
ax.set_xticklabels(pt.index)
plt.legend()
plt.title("How often have people votes Yes or No?")
plt.show()

Get the y value of a given x

I have a simple question but have not found any answer..
Let's have a look at this code :
from matplotlib import pyplot
import numpy
x=[0,1,2,3,4]
y=[5,3,40,20,1]
pyplot.plot(x,y)
It is plotted and all the points ared linked.
Let's say I want to get the y value of x=1,3.
How can I get the x values matching with y=30 ? (there are two)
Many thanks for your help
You could use shapely to find the intersections:
import matplotlib.pyplot as plt
import numpy as np
import shapely.geometry as SG
x=[0,1,2,3,4]
y=[5,3,40,20,1]
line = SG.LineString(list(zip(x,y)))
y0 = 30
yline = SG.LineString([(min(x), y0), (max(x), y0)])
coords = np.array(line.intersection(yline))
print(coords[:, 0])
fig, ax = plt.subplots()
ax.axhline(y=y0, color='k', linestyle='--')
ax.plot(x, y, 'b-')
ax.scatter(coords[:, 0], coords[:, 1], s=50, c='red')
plt.show()
finds solutions for x at:
[ 1.72972973 2.5 ]
The following code might do what you want. The interpolation of y(x) is straight forward, as the x-values are monotonically increasing. The problem of finding the x-values for a given y is not so easy anymore, once the function is not monotonically increasing as in this case. So you still need to know roughly where to expect the values to be.
import numpy as np
import scipy.interpolate
import scipy.optimize
x=np.array([0,1,2,3,4])
y=np.array([5,3,40,20,1])
#if the independent variable is monotonically increasing
print np.interp(1.3, x, y)
# if not, as in the case of finding x(y) here,
# we need to find the zeros of an interpolating function
y0 = 30.
initial_guess = 1.5 #for the first zero,
#initial_guess = 3.0 # for the secon zero
f = scipy.interpolate.interp1d(x,y,kind="linear")
fmin = lambda x: np.abs(f(x)-y0)
s = scipy.optimize.fmin(fmin, initial_guess, disp=False)
print s
I use python 3.
print(numpy.interp(1.3, x, y))
Y = 30
eps = 1e-6
j = 0
for i, ((x0, x1), (y0, y1)) in enumerate(zip(zip(x[:-1], x[1:]), zip(y[:-1], y[1:]))):
dy = y1 - y0
if abs(dy) < eps:
if y0 == Y:
print('There are infinite number of solutions')
else:
t = (Y - y0)/dy
if 0 < t < 1:
sol = x0 + (x1 - x0)*t
print('solution #{}: {}'.format(j, sol))
j += 1

Matplotlib darker hsv colormap

I'm using the HSV colormap from matplotlib to plot some vector fields. Is there a way to darken or make smoother the HSV colours so they look more like this
than my original plot colours, which are too bright:
Introduction
Assuming you're trying to plot a pcolor image like this:
import numpy as np
import matplotlib.pyplot as plt
y, x = np.mgrid[slice(-3, 3 + 0.05, 0.05),
slice(-3, 3 + 0.15, 0.15)]
z = (1 - x / 2. + x ** 5 + y ** 3) * np.exp(-x ** 2 - y ** 2)
# x and y are bounds, so z should be the value *inside* those bounds.
# Therefore, remove the last value from the z array.
z = z[:-1, :-1]
fig = plt.figure(1)
fig.clf()
ax = plt.gca()
pcol = ax.pcolormesh(x, y, z, cmap=plt.get_cmap('hsv'), )
plt.colorbar(pcol)
ax.set_xlim([-3, 3])
ax.set_ylim([-3, 3])
Your image will be:
Methods
I've written an alternate implementation of the MPL cookbook cmap_map function that modifies colormaps. In addition to support for kwargs and pep8 compliance, this version handles discontinuities in a colormap:
import numpy as np
from matplotlib.colors import LinearSegmentedColormap as lsc
def cmap_map(function, cmap, name='colormap_mod', N=None, gamma=None):
"""
Modify a colormap using `function` which must operate on 3-element
arrays of [r, g, b] values.
You may specify the number of colors, `N`, and the opacity, `gamma`,
value of the returned colormap. These values default to the ones in
the input `cmap`.
You may also specify a `name` for the colormap, so that it can be
loaded using plt.get_cmap(name).
"""
if N is None:
N = cmap.N
if gamma is None:
gamma = cmap._gamma
cdict = cmap._segmentdata
# Cast the steps into lists:
step_dict = {key: map(lambda x: x[0], cdict[key]) for key in cdict}
# Now get the unique steps (first column of the arrays):
step_list = np.unique(sum(step_dict.values(), []))
# 'y0', 'y1' are as defined in LinearSegmentedColormap docstring:
y0 = cmap(step_list)[:, :3]
y1 = y0.copy()[:, :3]
# Go back to catch the discontinuities, and place them into y0, y1
for iclr, key in enumerate(['red', 'green', 'blue']):
for istp, step in enumerate(step_list):
try:
ind = step_dict[key].index(step)
except ValueError:
# This step is not in this color
continue
y0[istp, iclr] = cdict[key][ind][1]
y1[istp, iclr] = cdict[key][ind][2]
# Map the colors to their new values:
y0 = np.array(map(function, y0))
y1 = np.array(map(function, y1))
# Build the new colormap (overwriting step_dict):
for iclr, clr in enumerate(['red', 'green', 'blue']):
step_dict[clr] = np.vstack((step_list, y0[:, iclr], y1[:, iclr])).T
return lsc(name, step_dict, N=N, gamma=gamma)
Implementation
To use it, simply define a function that will modify your RGB colors as you like (values from 0 to 1) and supply it as input to cmap_map. To get colors close to the ones in the images you provided, for example, you could define:
def darken(x, ):
return x * 0.8
dark_hsv = cmap_map(darken, plt.get_cmap('hsv'))
And then modify the call to pcolormesh:
pcol = ax.pcolormesh(x, y, z, cmap=dark_hsv)
If you only wanted to darken the greens in the image, you could do (now all in one line):
pcol = ax.pcolormesh(x, y, z,
cmap=cmap_map(lambda x: x * [1, 0.7, 1],
plt.get_cmap('hsv'))
)