I am a begineer in plotting graphs in bokeh. So please forgive me if this is a stupid question.
I am trying to plot a line grpah, where my data is in a dataframe and I have provided the x and y axis as lists.
But some of my data in y axis has nonetype objects in it.
when it is nonetype in "datapoints" column the corresponding "datapoint_count" has a list like [1]. Otherwise the "dataponts" colums dhould have a list of 20 floats and corresponding datapoint_count column should have a list of 1-20 digits.
So basically I want the x axis of the graph to show a range of 1-20y axis should plot the datapoints whichill range between 90.0 - 180.0
When I am running the code there is no python error but if I go to the browser and check developer's tool it says that the bokeh could not set initial ranges.
data=df
random_figure = figure(title='random', x_axis_label="Index", y_axis_label="random [ms]",
plot_width=800, plot_height=400, output_backend="webgl")
random_figure.add_tools(random_hover)
id_values = data['testcase_id'].drop_duplicates()
data_temp= data[['id', 'datapoints']].copy()
data_temp['datapoint_count'] = None
data_temp['datapoint_count'] = data_temp['datapoint_count'].astype(object)
for indexes, item in data_temp.iterrows():
if item['datapoints'] is None or str(item['datapoints']) == '[]': # this has nonetype or strings
item['datapoints'] = [0]
else:
item['datapoints'] = [float(x) for x in item['datapoints'].strip('[').strip(']').split(',')]
iter_nr = 0
raw_data_count = []
for each in item['datapoints']:
iter_nr += 1
datapoint_count.append(iter_nr)
data_temp.at[indexes, 'datapoint_count'] = datapoint_count
name_dict_random = {'name': [], 'legend': [], 'label': []}
logging.info('START OF DRAWINGS')
for ind, id in enumerate(id_values):
it_color = Turbo256[random.randint(0, 255)]
name_glyph_random = random_figure.line(x='datapoint_count',
y='datapoints',
line_width=2,
legend_label=str(id),
source=data_temp.where(
data_temp['id'] == id).dropna(),
color=it_color)
name_dict_random['name'].append(name_glyph_random)
name_dict_random['label'].append(str(id))
logging.info('AFTER DRAWINGS LOOP')
for label in range(len(data.id.unique())):
name_dict_random['legend'].append(random_figure.legend.items[label])
initial_value = []
options = list(data.id.unique())
for i, name in enumerate(options):
options[i] = str(name)
for i in range(len(options)):
if name_dict_random['label'][i] in initial_value:
name_dict_random['name'][i].visible = True
name_dict_random['legend'][i].visible = True
else:
name_dict_random['name'][i].visible = False
name_dict_random['legend'][i].visible = False
I have solved it now.
Actually though the dataframe showed that the rows content arrays they were actually categorized as objects.
Bokeh could not understand what to do with object in the axis.
So now I have referred them wih iloc:
x = data[data['id'] == id]['datapoint_count'].iloc[0]
y = data[data['id'] == id]['datapoint'].iloc[0]
name_glyph_handover = handover_figure.line(x=x, y=y, line_width=2,
legend_label=str(id), color=it_color)
Related
In my program, im using mplcursors on a matplotlib graph so I can identify certain points precisely.
mplcursors.cursor(multiple=True).connect("add", lambda sel: sel.annotation.draggable(False))
Now I made a complex graph with multiple axis:
first = 1
offset = 60
for x in range(len(cat_list)):
if "Time" not in cat_list[x]:
if first and not cat_list[x].startswith("EngineSpeed"):
parasites[x] = ParasiteAxes(host, sharex = host)
host.parasites.append(parasites[x])
parasites[x].axis["right"].set_visible(True)
parasites[x].set_ylabel(cat_list[x])
parasites[x].axis["right"].major_ticklabels.set_visible(True)
parasites[x].axis["right"].label.set_visible(True)
p_plot, = parasites[x].plot(t, t_num_list[x], label = cat_list[x])
#parasites[x].axis["right"+str(x+1)].label.set_color(p_plot.get_color())
parasites[x].axis["right"].label.set_color(p_plot.get_color())
first = 0
elif not cat_list[x].startswith("EngineSpeed"):
parasites[x] = ParasiteAxes(host, sharex = host)
host.parasites.append(parasites[x])
parasites[x].set_ylabel(cat_list[x])
new_axisline = parasites[x].get_grid_helper().new_fixed_axis
parasites[x].axis["right"+str(x+1)] = new_axisline(loc = "right",
axes = parasites[x],
offset = (offset, 0))
p_plot, = parasites[x].plot(t, t_num_list[x])
parasites[x].axis["right"+str(x+1)].label.set_color(p_plot.get_color())
offset = offset + 60
host.legend()
fig.add_axes(host)
plt.show()
This code results in the following graph:
https://i.stack.imgur.com/Wl7yC.png
Now I have to somehow be able to select certain points by selecting which axis im using. How do I make a selection menu for choosing an active axis and how do I then use mplcursors to select my points?
Thanks,
Ziga
I am working on how to use KNN to predict a rating for a movie. I use a video and a book to teach myself how to go about it
I tried to run the code I found in the book but it gave me error message. I googled the error message so as to understand it and fix my problem but I don't think I know how to adapt the solutions to my problem.
import numpy as np
import pandas as pd
r_cols = ['user_id', 'movie_id', 'rating']
ratings = pd.read_csv('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.data', sep='\t', engine='python', names=r_cols, usecols=range(3)) # please enter your file path here. The file is u.data
print(ratings.head())
movieProperties = ratings.groupby('movie_id').agg({'rating': [np.size, np.mean]})
print(movieProperties.head())
movieNumRatings = pd.DataFrame(movieProperties['rating']['size'])
movieNormalizedNumRatings = movieNumRatings.apply(lambda x: (x - np.min(x)) / (np.max(x) - np.min(x)))
print(movieNormalizedNumRatings.head())
movieDict = {}
with open('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.item') as f: # The file is u.item
temp = ''
for line in f:
fields = line.rstrip('\n').split('|')
movieID = int(fields[0])
name = fields[1]
genres = fields[5:25]
genres = map(int, genres)
movieDict[movieID] = (name, genres, movieNormalizedNumRatings.loc[movieID].get('size'), movieProperties.loc[movieID].rating.get('mean'))
print(movieDict[1])
from scipy import spatial
def ComputeDistance(a, b):
genresA = np.array(list(a[1]))
genresB = np.array(list(b[1]))
genreDistance = spatial.distance.cosine(genresA, genresB)
popularityA = np.array(a[2])
popularityB = np.array(b[2])
popularityDistance = abs(popularityA - popularityB)
return genreDistance + popularityDistance
print(ComputeDistance(movieDict[2], movieDict[4]))
import operator
def getNeighbors(movieID, K):
distances = []
for movie in movieDict:
if (movie != movieID):
dist = ComputeDistance(movieDict[movieID], movieDict[movie])
distances.append((movie, dist))
distances.sort(key=operator.itemgetter(1))
neighbors = []
for x in range(K):
neighbors.append(distance[x][0])
return neighbors
K = 10
avgRating = 0
neighbors = getNeighbors(1, K)
I got this error message from PowerShell:
Traceback(most recent call last):
neighbors = getNeighbors(1, K)
dist = ComputeDistance(movieDict[movieID], movieDict[movie])
genreDistance = spatial.distance.cosine(genresA, genresB)
return correlation(u, v, w=w, centered=False)
uv = np.average(u*v, weights=w)
ValueError: operands could not be broadcast together with shape (19,)(0,)
I got this error message when I tried to debug the problem from ipython terminal:
c:\programdata\anaconda3\lib\site-packages\scipy\spatial\distance.py(695)correlation()
693 u = u - umu
694 v = v - vmu
---> 695 uv = np.average(u*v, weights=w)
696 uu = np.average(np.square(u), weights=w)
697 vv = np.average(np.square(v), weights=w)
**Note**: The code ran fine and produced results up until *print(Cprint(ComputeDistance(movieDict[2], movieDict[4]))*
My guess is the problem is with this part of the code:
import operator
def getNeighbors(movieID, K):
distances = []
for movie in movieDict:
if (movie != movieID):
dist = ComputeDistance(movieDict[movieID], movieDict[movie])
distances.append((movie, dist))
distances.sort(key=operator.itemgetter(1))
neighbors = []
for x in range(K):
neighbors.append(distance[x][0])
return neighbors
K = 10
avgRating = 0
neighbors = getNeighbors(1, K)
The code can be found in this link: https://hendra-herviawan.github.io/Movie-Recommendation-based-on-KNN-K-Nearest-Neighbors.html
The error of "operands could not be broadcast together with shape (x,)(y,)" usually raises when you are trying to perform an operation between two arrays that must have the same shape but they don't. In your case you are trying to take an weighted average between two arrays u and v. The arrays u and v don't have the length.
I saw that you parsing a movies list by splitting the lines with the "|" character and then storing these results in a dictionary. Probably this file or its division with "|" are returning different results.
The error log shows that the second array doesn't have any element, this could be generated by an empty line on the movies files.
I'm trying some exercise.
I looked for this problem before, but didn't find one for my problem.
This code seems to work with trainX, but not with trainY.
I have 1672 data for trainY in 1D for one neuron output.
batch_dim = trainX.shape[0]
input_dim = windowSize
hidden_dim = 6
output_dim = 1
O: batch_dim=1 with value "1672"
X = trainX[index:index+batch_dim,:]
Y = trainY[index:index+batch_dim,:]
index = index+batch_dim
The problem seems to be in the dimension. So I try to reshape it
Y = np.reshape(trainY[index:index+batch_dim,:],-1,1)
but it doesn't solve anything. The output still work, but error still there.
I just wanted the error to go away.
The variable size output:
batch_dim = 1 (value = 1672)
index = 1 (value = 0)
X : (1672,3)
Y : (1672,)
Y = trainY[index:index+batch_dim,:]
IndexError: too many indices for array
I am building machine learning models for a certain data set. Then, based on the constraints and bounds for the outputs and inputs, I am trying to find the input parameters for the most minimized answer.
The problem which I am facing is that, when the model is a linear regression model or something like lasso, the minimization works perfectly fine.
However, when the model is "Decision Tree", it constantly returns the very initial value that is given to it. So basically, it does not enforce the constraints.
import numpy as np
import pandas as pd
from scipy.optimize import minimize
I am using the very first sample from the input data set for the optimization. As it is only one sample, I need to reshape it to (1,-1) as well.
x = df_in.iloc[0,:]
x = np.array(x)
x = x.reshape(1,-1)
This is my Objective function:
def objective(x):
x = np.array(x)
x = x.reshape(1,-1)
y = 0
for n in range(df_out.shape[1]):
y = Model[n].predict(x)
Y = y[0]
return Y
Here I am defining the bounds of inputs:
range_max = pd.DataFrame(range_max)
range_min = pd.DataFrame(range_min)
B_max=[]
B_min =[]
for i in range(range_max.shape[0]):
b_max = range_max.iloc[i]
b_min = range_min.iloc[i]
B_max.append(b_max)
B_min.append(b_min)
B_max = pd.DataFrame(B_max)
B_min = pd.DataFrame(B_min)
bnds = pd.concat([B_min, B_max], axis=1)
These are my constraints:
con_min = pd.DataFrame(c_min)
con_max = pd.DataFrame(c_max)
Here I am defining the constraint function:
def const(x):
x = np.array(x)
x = x.reshape(1,-1)
Y = []
for n in range(df_out.shape[1]):
y = Model[n].predict(x)[0]
Y.append(y)
Y = pd.DataFrame(Y)
a4 =[]
for k in range(Y.shape[0]):
a1 = Y.iloc[k,0] - con_min.iloc[k,0]
a2 = con_max.iloc[k, 0] - Y.iloc[k,0]
a3 = [a2,a1]
a4 = np.concatenate([a4, a3])
return a4
c = const(x)
con = {'type': 'ineq', 'fun': const}
This is where I try to minimize. I do not pick a method as the automatically picked model has worked so far.
sol = minimize(fun = objective, x0=x,constraints=con, bounds=bnds)
So the actual constraints are:
c_min = [0.20,1000]
c_max = [0.3,1600]
and the max and min range for the boundaries are:
range_max = [285,200,8,85,0.04,1.6,10,3.5,20,-5]
range_min = [215,170,-1,60,0,1,6,2.5,16,-18]
I think you should check the output of 'sol'. At times, the algorithm is not able to perform line search completely. To check for this, you should check message associated with 'sol'. In such a case, the optimizer returns initial parameters itself. There may be various reasons of this behavior. In a nutshell, please check the output of sol and act accordingly.
Arad,
If you have not yet resolved your issue, try using scipy.optimize.differential_evolution instead of scipy.optimize.minimize. I ran into similar issues, particularly with decision trees because of their step-like behavior resulting in infinite gradients.
I asked this question yesterday about storing a plot within an object. I tried implementing the first approach (aware that I did not specify that I was using qplot() in my original question) and noticed that it did not work as expected.
library(ggplot2) # add ggplot2
string = "C:/example.pdf" # Setup pdf
pdf(string,height=6,width=9)
x_range <- range(1,50) # Specify Range
# Create a list to hold the plot objects.
pltList <- list()
pltList[]
for(i in 1 : 16){
# Organise data
y = (1:50) * i * 1000 # Get y col
x = (1:50) # get x col
y = log(y) # Use natural log
# Regression
lm.0 = lm(formula = y ~ x) # make linear model
inter = summary(lm.0)$coefficients[1,1] # Get intercept
slop = summary(lm.0)$coefficients[2,1] # Get slope
# Make plot name
pltName <- paste( 'a', i, sep = '' )
# make plot object
p <- qplot(
x, y,
xlab = "Radius [km]",
ylab = "Services [log]",
xlim = x_range,
main = paste("Sample",i)
) + geom_abline(intercept = inter, slope = slop, colour = "red", size = 1)
print(p)
pltList[[pltName]] = p
}
# close the PDF file
dev.off()
I have used sample numbers in this case so the code runs if it is just copied. I did spend a few hours puzzling over this but I cannot figure out what is going wrong. It writes the first set of pdfs without problem, so I have 16 pdfs with the correct plots.
Then when I use this piece of code:
string = "C:/test_tabloid.pdf"
pdf(string, height = 11, width = 17)
grid.newpage()
pushViewport( viewport( layout = grid.layout(3, 3) ) )
vplayout <- function(x, y){viewport(layout.pos.row = x, layout.pos.col = y)}
counter = 1
# Page 1
for (i in 1:3){
for (j in 1:3){
pltName <- paste( 'a', counter, sep = '' )
print( pltList[[pltName]], vp = vplayout(i,j) )
counter = counter + 1
}
}
dev.off()
the result I get is the last linear model line (abline) on every graph, but the data does not change. When I check my list of plots, it seems that all of them become overwritten by the most recent plot (with the exception of the abline object).
A less important secondary question was how to generate a muli-page pdf with several plots on each page, but the main goal of my code was to store the plots in a list that I could access at a later date.
Ok, so if your plot command is changed to
p <- qplot(data = data.frame(x = x, y = y),
x, y,
xlab = "Radius [km]",
ylab = "Services [log]",
xlim = x_range,
ylim = c(0,10),
main = paste("Sample",i)
) + geom_abline(intercept = inter, slope = slop, colour = "red", size = 1)
then everything works as expected. Here's what I suspect is happening (although Hadley could probably clarify things). When ggplot2 "saves" the data, what it actually does is save a data frame, and the names of the parameters. So for the command as I have given it, you get
> summary(pltList[["a1"]])
data: x, y [50x2]
mapping: x = x, y = y
scales: x, y
faceting: facet_grid(. ~ ., FALSE)
-----------------------------------
geom_point:
stat_identity:
position_identity: (width = NULL, height = NULL)
mapping: group = 1
geom_abline: colour = red, size = 1
stat_abline: intercept = 2.55595281266726, slope = 0.05543539319091
position_identity: (width = NULL, height = NULL)
However, if you don't specify a data parameter in qplot, all the variables get evaluated in the current scope, because there is no attached (read: saved) data frame.
data: [0x0]
mapping: x = x, y = y
scales: x, y
faceting: facet_grid(. ~ ., FALSE)
-----------------------------------
geom_point:
stat_identity:
position_identity: (width = NULL, height = NULL)
mapping: group = 1
geom_abline: colour = red, size = 1
stat_abline: intercept = 2.55595281266726, slope = 0.05543539319091
position_identity: (width = NULL, height = NULL)
So when the plot is generated the second time around, rather than using the original values, it uses the current values of x and y.
I think you should use the data argument in qplot, i.e., store your vectors in a data frame.
See Hadley's book, Section 4.4:
The restriction on the data is simple: it must be a data frame. This is restrictive, and unlike other graphics packages in R. Lattice functions can take an optional data frame or use vectors directly from the global environment. ...
The data is stored in the plot object as a copy, not a reference. This has two
important consequences: if your data changes, the plot will not; and ggplot2 objects are entirely self-contained so that they can be save()d to disk and later load()ed and plotted without needing anything else from that session.
There is a bug in your code concerning list subscripting. It should be
pltList[[pltName]]
not
pltList[pltName]
Note:
class(pltList[1])
[1] "list"
pltList[1] is a list containing the first element of pltList.
class(pltList[[1]])
[1] "ggplot"
pltList[[1]] is the first element of pltList.
For your second question: Multi-page pdfs are easy -- see help(pdf):
onefile: logical: if true (the default) allow multiple figures in one
file. If false, generate a file with name containing the
page number for each page. Defaults to ‘TRUE’.
For your main question, I don't understand if you want to store the plot inputs in a list for later processing, or the plot outputs. If it is the latter, I am not sure that plot() returns an object you can store and retrieve.
Another suggestion regarding your second question would be to use either Sweave or Brew as they will give you complete control over how you display your multi-page pdf.
Have a look at this related question.