I presume it is some kind of moving average, but the valid range is between 0 and 1.
It is called exponential moving average, below is a code explanation how it is created.
Assuming all the real scalar values are in a list called scalars the smoothing is applied as follows:
def smooth(scalars: List[float], weight: float) -> List[float]: # Weight between 0 and 1
last = scalars[0] # First value in the plot (first timestep)
smoothed = list()
for point in scalars:
smoothed_val = last * weight + (1 - weight) * point # Calculate smoothed value
smoothed.append(smoothed_val) # Save it
last = smoothed_val # Anchor the last smoothed value
return smoothed
Here is the actual piece of source code that performs that exponential smoothing the with some additional de-biasing explained in the comments to compensate for the choice of the zero initial value:
last = last * smoothingWeight + (1 - smoothingWeight) * nextVal
Source: https://github.com/tensorflow/tensorboard/blob/34877f15153e1a2087316b9952c931807a122aa7/tensorboard/components/vz_line_chart2/line-chart.ts#L714
The implementation of EMA smoothing used for TensorBoard can be found here.
The equivalent in Python is actually:
def smooth(scalars: list[float], weight: float) -> list[float]:
"""
EMA implementation according to
https://github.com/tensorflow/tensorboard/blob/34877f15153e1a2087316b9952c931807a122aa7/tensorboard/components/vz_line_chart2/line-chart.ts#L699
"""
last = 0
smoothed = []
num_acc = 0
for next_val in scalars:
last = last * weight + (1 - weight) * next_val
num_acc += 1
# de-bias
debias_weight = 1
if weight != 1:
debias_weight = 1 - math.pow(weight, num_acc)
smoothed_val = last / debias_weight
smoothed.append(smoothed_val)
return smoothed
Related
I'm plotting a facebook-prophet forecast and I only want to plot the last month data. This 'last month' changes from month to month so I can't use plt.xlim() to limit it by a given range.
Is there any way to limit the x-axis by a given number of elements, like plot last 1000 x-axis values no matter what those values are?
I'd say write a function that represents your understanding of the appropriate x limits. For example:
def get_x_limits(data):
lower_x = min(data)
# or
lower_x = min(data)*0.9 # 10% below lowest value (if values are positive)
# or
lower_x = min(data) - 1 # a little below lowest value
# similarly
upper_x = max(data)
# or
upper_x = max(data)*1.1 # 10% above highest value (if values are positive)
# or
upper_x = max(data) + 1 # a little above highest value
# maybe smth like this
if upper_x - lower_x < 1000:
upper_x = lower_x + 1000
# and finally
return lower_x, upper_x
You can then use these values to set your limits:
lower_x, upper_x = get_x_limits(data)
plt.xlim(lower_x,upper_x)
I have to admit that I ignored the question for the number of elements since I thought its mostly relevant what's in your data, not how much data you have. However, you can still work len(data) into the get_x_limits function the way it fits your need.
I performed spherical Kriging, but I can't seem to get good output graphs.
The coordinates(x, and y) range from around around 51 latitude and around 6.5 as longitude
my observations range from -70 to +10
here is my code :
import openturns as ot
import pandas as pd
# your input / output data can be easily formatted as samples for openturns
df = pd.read_csv("kreuzkerpenutm.csv")
inputdata = ot.Sample(df[['x','y']].values)
outputdata = ot.Sample(df[['z']].values)
dimension = 2 # dimension of your input (x,y)
basis = ot.ConstantBasisFactory(dimension).build()
covarianceModel = ot.SphericalModel(dimension)
algo = ot.KrigingAlgorithm(inputdata, outputdata, covarianceModel, basis)
algo.run()
result = algo.getResult()
metamodel = result.getMetaModel()
lower = [-10.0] * 2 # lower bound of the 2D window
upper = [50.0] * 2 # upper bound of the 2D window
graph = metamodel.draw(lower, upper)
graph.setBoundingBox(ot.Interval(lower, upper))
graph.add(ot.Cloud(inputdata)) # overlay a scatter plot of the observation points
graph.setTitle("Kriging metamodel")
# A View object allows us to interact with the underlying matplotlib figure
from openturns.viewer import View
view = View(graph, legend_kw={'bbox_to_anchor':(1,1), 'loc':"upper left"})
view.getFigure().tight_layout()
here is my output:
kriging metamodel graph
I don't know why my graph won't show me my inputs aswell as my kriging results.
thanks for ideas and help
If the input data is not scaled in [-1,1]^d, the kriging metamodel may have issues to identify the scale parameters using maximum likelihood optimization. In order to help for this, we may:
provide a better starting point for the scale parameters of the covariance model (this is trick "A" below),
set the bounds of the optimization algorithm so that the interval where the parameters are searched for correspond to the data at hand (this is trick "B" below).
This is what the following script does, using simulated data instead of a csv data file. In the script, I create the data using a g function which is scaled so that it produces results in the [-10, 70] range, as in your problem. Please look carefuly at the setScale() method which sets the initial value of the covariance model: this is the starting point of the optimization algorithm. Then look at the setOptimizationBounds() method, which sets the bounds of the optimization algorithm.
import openturns as ot
dimension = 2 # dimension of your input (x,y)
distribution = ot.ComposedDistribution([ot.Uniform(-10.0, 50.0)] * dimension)
inputdata = distribution.getSample(100)
g = ot.SymbolicFunction(["x", "y"], ["30 + 3.0 * sin(x / 10.0) * (y / 10.0) ^ 2"])
outputdata = g(inputdata)
basis = ot.ConstantBasisFactory(dimension).build()
covarianceModel = ot.SphericalModel(dimension)
covarianceModel.setScale(inputdata.getMax()) # Trick A
algo = ot.KrigingAlgorithm(inputdata, outputdata, covarianceModel, basis)
# Trick B, v2
x_range = inputdata.getMax() - inputdata.getMin()
scale_max_factor = 2.0 # Must be > 1, tune this to match your problem
scale_min_factor = 0.1 # Must be < 1, tune this to match your problem
maximum_scale_bounds = scale_max_factor * x_range
minimum_scale_bounds = scale_min_factor * x_range
scaleOptimizationBounds = ot.Interval(minimum_scale_bounds, maximum_scale_bounds)
algo.setOptimizationBounds(scaleOptimizationBounds)
algo.run()
result = algo.getResult()
metamodel = result.getMetaModel()
metamodel.setInputDescription(["x", "y"])
metamodel.setOutputDescription(["z"])
lower = [-10.0] * 2 # lower bound of the 2D window
upper = [50.0] * 2 # upper bound of the 2D window
graph = metamodel.draw(lower, upper)
graph.setBoundingBox(ot.Interval(lower, upper))
graph.add(ot.Cloud(inputdata)) # overlay a scatter plot of the observation points
graph.setTitle("Kriging metamodel")
# A View object allows us to interact with the underlying matplotlib figure
from openturns.viewer import View
view = View(graph, legend_kw={"bbox_to_anchor": (1, 1), "loc": "upper left"})
view.getFigure().tight_layout()
The previous script produces the following figure.
There are other ways to implement trick B. Here is one provided by J.Pelamatti:
# Trick B, v3
for d in range(X_train.getDimension()):
dist = scipy.spatial.distance.pdist(X_train[:,d])
scale_max_factor = 2.0 # Must be > 1, tune this to match your problem
scale_min_factor = 0.1 # Must be < 1, tune this to match your problem
maximum_scale_bounds = scale_max_factor * np.max(dist)
minimum_scale_bounds = scale_min_factor * np.min(dist)
This topic is discussed in this particular thread in OT's forum.
Sorry for the late answer.
Which version of openturns are you using?
Probably you have an embedded transformation of (input) data, which makes the data range between (-3, 3) approximately (standard scaling). The kriging result should contains the transformation in such a case.
With more recent openturns implementations, this feature has been removed.
Hope this can help.
Cheers
I am using this function to calculate distance between 2 vectors a,b, of size 300, word2vec, I get the distance between 'hot' and 'cold' to be equal 1.
How to add this value (1) to a vector, becz i thought simply new_vec=model['hot']+1, but when I do the calc dist(new_vec,model['hot'])=17?
import numpy
def dist(a,b):
return numpy.linalg.norm(a-b)
a=model['hot']
c=a+1
dist(a,c)
17
I expected dist(a,c) will give me back 1!
You should review what the norm is. In the case of numpy, the default is to use the L-2 norm (a.k.a the Euclidean norm). When you add 1 to a vector, the call is to add 1 to all of the elements in the vector.
>> vec1 = np.random.normal(0,1,size=300)
>> print(vec1[:5])
... [ 1.18469795 0.04074346 -1.77579852 0.23806222 0.81620881]
>> vec2 = vec1 + 1
>> print(vec2[:5])
... [ 2.18469795 1.04074346 -0.77579852 1.23806222 1.81620881]
Now, your call to norm is saying sqrt( (a1-b1)**2 + (a2-b2)**2 + ... + (aN-bN)**2 ) where N is the length of the vector and a is the first vector and b is the second vector (and ai being the ith element in a). Since (a1-b1)**2 == (a2-b2)**2 == ... == (aN-bN)**2 == 1 we expect this sum to produce N which in your case is 300. So sqrt(300) = 17.3 is the expected answer.
>> print(np.linalg.norm(vec1-vec2))
... 17.320508075688775
To answer the question, "How to add a value to a vector": you have done this correctly. If you'd like to add a value to a specific element then you can do vec2[ix] += value where ix indexes the element that you wish to add. If you want to add a value uniformly across all elements in the vector that will change the norm by 1, then add np.sqrt(1/300).
Also possibly relevant is a more commonly used distance metric for word2vec vectors: the cosine distance which measures the angle between two vectors.
I am implementing a very simple animation effect for a game. The scenario is like this:
there is a elastic rubber line, length is 1 meter, when it is extended over 1 meter, it is elastic.
the line connects two dots A and B like this, the distance is S, S > 1 meter
A <------------- B
then fix dot A, and releases B, the line takes B to the direction of A
I want to know how to calculate time T, which B costs to move X meters towards A (X <= S).
Any ideas?
Thanks!
I have been meaning to learn how to animate these kinds of images in sage (a python based platform for math) for a while, so i used this as an excuse. I hope this code snippet and image is helpful.
A = 3
w = 0.5
# x = f(t) = A cos(wt) inside elastic region
# with x = displacement from 1 meter mark
# in the below code, x is the displacement from origin (x = A cos(wt) + 1)
# find speed when we cross the one meter mark
# f'(t) = -Aw sin(wt), but this is also max speed
# ie f'(t at one meter mark) = -Aw
speed_max = -A * w
# time to reach max speed + time to cross last meter
eta = float(pi/2 * 1/w + 1/abs(speed_max))
# the function you were looking for
def time_left(x):
if x < 1:
return x/abs(speed_max)
else:
return 1/w * arccos((x-1)/A)
It may not be clear in the image but within one meter of the origin there is no acceleration.
I am trying to write an algorithm in python for knapsack problem. I did quite a few iterations and came to the following solution. It seems perfect for me. When I ran it on test sets,
it does not output optimal value
and sometimes gives maximum recursion depth error.
after changing the maxValues function it is outputting the optimal value, But it takes very very long time for datasets having more points. how to refine it
For the second problem, I have inspected the data for which it gives the error. The data is like huge and only just couple of them exceeds and the knapsack capacity. So it unnecessarily goes through the entire list.
So what I planned to do is at the start of running my recursive function, I tried to see the entire weights list where each weight is less than the current capacity and prune the rest. The following is the code I am planning to implement.
#~ weights_jump = find_indices(temp_w, lambda e: e < capacity)
#~ if(len(weights_jump)>0):
#~ temp_w[0:weights_jump[0]-1] = []
#~ temp_v[0:weights_jump[0]-1] = []
My main problem remains that why it is not outputting the optimal value. Please help me in this regards and also to integrate the above code into the current algorithm
The following is the main function. the input for this function is as follows,
A knapsack input contains n + 1 lines. The first line contains two integers, the first is the number
of items in the problem, n. The second number is the capacity of the knapsack, K. The remaining
lines present the data for each of the items. Each line, i ∈ 0 . . . n − 1 contains two integers, the
item’s value vi followed by its weight wi
eg input:
n K
v_0 w_0
v_1 w_1
...
v_n-1 w_n-1
def solveIt(inputData):
# parse the input
lines = inputData.split('\n')
firstLine = lines[0].split()
items = int(firstLine[0])
capacity = int(firstLine[1])
K = capacity
values = []
weights = []
for i in range(1, items+1):
line = lines[i]
parts = line.split()
values.append(int(parts[0]))
weights.append(int(parts[1]))
items = len(values)
#end of parsing
value = 0
weight = 0
print(weights)
print(values)
v = node(value,weights,values,K,0,taken);
# prepare the solution in the specified output format
outputData = str(v[0]) + ' ' + str(0) + '\n'
outputData += ' '.join(map(str, v[1]))
return outputData
The following is the recursive function
I'will try to explain this recursive function. Let's say I 'm starting off with root node and now I ill have two decisions to make either to take the first element or not.
before this I will call maxValue function to see the maximum value that can be obtained following this branch. If it is less than existing_max no need to search, so prune.
i will follow left branch if the weight of the first element is less than capacity. so append(1).
updating values,weights list etc and again call node function.
so it first transverses entire left branch and then transverses right branch.
in the right im just updating the values,weights lists and calling node function.
For this function inputs are
value -- The current value of the problem, It is initially set to zero and is it goes it gets increased
weights list
values list
current capacity
current max value found by algo. If this existing_max is greater than the maximum value that can be obtained by following a branch,
there is no need to search that branch. so entire branch is pruned
existing_nodes is the list which tells whether a particular item is taken (1) or not (0)
def node(value,weights,values,capacity,existing_max,existing_nodes):
v1=[];e1=[]; #values we get from left branch
v2=[];e2=[]; #values we get from right branch
e=[];
e = existing_nodes[:];
temp_w = weights[:]
temp_v = values[:];
#first check if the list is empty
if(len(values)==0):
r = [value,existing_nodes[:]]
return r;
#centre check if this entire branch could be pruned. it checks for max value that can be obtained is more than the max value inputted to this
max_value = value+maxValue(weights,values,capacity);
print('existing _max is '+str(existing_max))
print('weight in concern '+str(weights[0])+' value is '+str(value))
if(max_value<=existing_max):
return [0,[]];
#Transversing the left branch
#Transverse only if the weight does not exceed the capacity
print colored('leftbranch','red');
#search for indices of weights where weight < capacity
#~ weights_jump = find_indices(temp_w, lambda e: e < capacity)
#~ if(len(weights_jump)>0):
#~ temp_w[0:weights_jump[0]-1] = []
#~ temp_v[0:weights_jump[0]-1] = []
if(temp_w[0]<=capacity):
updated_value = temp_v[0]+value;
k = capacity-temp_w[0];
temp_w.pop(0);
temp_v.pop(0);
e1 =e[:]
e1.append(1);
print(str(updated_value)+' '+str(k)+' ')
raw_input('press ')
v1= node(updated_value,temp_w,temp_v,k,existing_max,e1);
#after transversing left node update existing_max
if(v1[0]>existing_max):
existing_max = v1[0];
else:
v1 = [0,[]]
#Transverse the right branch
#it implies we are not including the current value so remove that from weights and values.
print('rightbranch')
#~ print(str(value)+' '+str(capacity)+' ')
raw_input("Press Enter to continue...")
weights.pop(0);
values.pop(0);
e2 =e[:];
e2.append(0);
v2 = node(value,weights,values,capacity,existing_max,e2);
if(v1[0]>v2[0]):
return v1;
else:
return v2;
The following is the helper function maxValue which is called from recursive function
def maxValues(weights,values,K):
weights = weights;
values = values;
a=[];
l = 0;
#~ print('hello');
items = len(weights);
#~ print(items);
max = 0;k = K;
for i in range(0,items):
t = (i,float(values[i])/weights[i]);
a.append(t);
#~ print(i);
a = sorted(a,key=operator.itemgetter(1),reverse=True);
#~ print(a);
length = len(a);
for (i,v) in a:
#~ print('this is loop'+str(l)+'and k is '+str(k)+'weight is '+str(weights[i]) );
if weights[i]<=k:
max = max+values[i];
w = weights[i];
#~ print('this'+str(w));
k = k-w;
if(k==0):
break;
else:
max = max+ (float(k)/weights[i])*values[i];
w = k
k = k-w;
#~ print('this is w '+str(max));
if(k==0):
break;
l= l+1;
return max;
I spent days on this and could not do anything.