Inconsistent results in Tensorflow incremental assignment with scatter_update - tensorflow

I am using Tensorflow's scatter_update to set the values of an array in a simple test. I would expect the following code to set all values in the array to 1.0.
pt = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
numOne = tf.constant(1)
xi = tf.Variable(0)
out = tf.Variable(pt)
tf.global_variables_initializer().run()
xi_ = xi + numOne
out_ = tf.scatter_update(out, [xi], [tf.cast(numOne, tf.float32)])
step = tf.group(
xi.assign(xi_),
out.assign(out_)
)
for i in range(15): step.run()
print(out.eval())
Instead I get inconsistent results like:
[ 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1.]
Is there some type of locking mechanism that I am missing?


Ops inside tf.group are executed in random order. If you want specific order, you could use tf.control_dependencies or split into two .run calls, ie
step1 = xi.assign(xi_)
step2 = out.assign(out_)
for i in range(15):
print(out.eval())
sess.run(step1)
sess.run(step2)


Also, using control_dependencies it would be something like:
xi_ = xi + numOne
with tf.control_dependencies[xi_]:
out_ = tf.scatter_update(out, [xi], [tf.cast(numOne, tf.float32)])
with tf.control_dependencies[out_]:
step = tf.group(
xi.assign(xi_),
out.assign(out_)
)
for i in range(15): step.run()
print(out.eval())

Related

Julia Jump : Getting all feasible solutions to mip

I would like to have instead of only the vector of optimal solution to a mip , all the feasible (suboptimal) vectors.
I found some old questions here, but I am not sure how they work.
First of all, is there any new library tool/way to do that automatically ?
I tried this but, it did nothing:
if termination_status(m) == MOI.FEASIBLE_POINT
println(x)
end
optimize!(m);
If not, what's the easiest way?
I thought of scanning the optimal solution till I find the first non -zero decision variable, then constraint this variable to be zero and solving the model again.
for i in 1:active_variables
if value.(z[i])==1
#constraint(m, x[i] == 0)
break
end
end
optimize!(m);
But I see this problem with this method** :
Ιf I constraint x[i] to be zero, in the next step I will want maybe to drop again this constraint? This comes down to whether there can exist two(or more) different solutions in which x[i]==1

JuMP supports returning multiple solutions.
Documentation: https://jump.dev/JuMP.jl/stable/manual/solutions/#Multiple-solutions
The workflow is something like:
using JuMP
model = Model()
#variable(model, x[1:10] >= 0)
# ... other constraints ...
optimize!(model)
if termination_status(model) != OPTIMAL
error("The model was not solved correctly.")
end
an_optimal_solution = value.(x; result = 1)
optimal_objective = objective_value(model; result = 1)
for i in 2:result_count(model)
#assert has_values(model; result = i)
println("Solution $(i) = ", value.(x; result = i))
obj = objective_value(model; result = i)
println("Objective $(i) = ", obj)
if isapprox(obj, optimal_objective; atol = 1e-8)
print("Solution $(i) is also optimal!")
end
end
But you need a solver that supports returning multiple solutions, and to configure the right solver-specific options.
See this blog post: https://jump.dev/tutorials/2021/11/02/tutorial-multi-jdf/

The following is an example of all-solution finder for a boolean problem. Such problems are easier to handle since the solution space is easily enumerated (even though it can still grow exponentially big).
First, let's get the packages and define the sample problem:
using Random, JuMP, HiGHS, MathOptInterface
function example_knapsack()
profit = [5, 3, 2, 7, 4]
weight = [2, 8, 4, 2, 5]
capacity = 10
minprofit = 10
model = Model(HiGHS.Optimizer)
set_silent(model)
#variable(model, x[1:5], Bin)
#objective(model, FEASIBILITY_SENSE, 0)
#constraint(model, weight' * x <= capacity)
#constraint(model, profit' * x >= minprofit)
return model
end
(it is a knapsack problem from the JuMP docs).
Next, we use recursion to explore the tree of all possible solutions. The tree does not go down branches with no solution (so the running time is not always exponential):
function findallsol(model, x)
perm = shuffle(1:length(x))
res = Vector{Float64}[]
_findallsol!(res, model, x, perm, 0)
return res
end
function _findallsol!(res, model, x, perm, depth)
n = length(x)
depth > n && return
optimize!(model)
if termination_status(model) == MathOptInterface.OPTIMAL
if depth == n
push!(res, value.(x))
return
else
idx = perm[depth+1]
v = value(x[idx])
newcon = #constraint(model, x[idx] == v)
_findallsol!(res, model, x, perm, depth + 1)
delete(model, newcon)
newcon = #constraint(model, x[idx] == 1 - v)
_findallsol!(res, model, x, perm, depth + 1)
delete(model, newcon)
end
end
return
end
Now we can:
julia> m = example_knapsack()
A JuMP Model
Maximization problem with:
Variables: 5
...
Names registered in the model: x
julia> res = findallsol(m, m.obj_dict[:x])
5-element Vector{Vector{Float64}}:
[1.0, 0.0, 0.0, 1.0, 1.0]
[0.0, 0.0, 0.0, 1.0, 1.0]
[1.0, 0.0, 1.0, 1.0, 0.0]
[1.0, 0.0, 0.0, 1.0, 0.0]
[0.0, 1.0, 0.0, 1.0, 0.0]
And we get a vector with all the solutions.
If the problem in question is a boolean problem, this method might be used, as is. In case it has non-boolean variables, the recursion will have to split the feasible space in some even fashion. For example, choosing a variable and cutting its domain in half, and recursing to each half with a smaller domain on this variable (to ensure termination).
P.S. This is not the optimal method. This problem has been well studied. Possible terms to search for are 'model counting' (especially in the boolean domain).
(UPDATE: Changed objective to use FEASIBLE)

Scipy curve_fit error, no optimal parameters

I am trying to use the scipy.optimize.curve_fit but I encounter an error - RuntimeError: Optimal parameters not found: Number of calls to function has reached maxfev = 800.
I have data points for the morse potential function and I am trying to extract the three parameters D, alpha and r_eq
from scipy.optimize import curve_fit
import numpy as np
import matplotlib.pyplot as plt
z_top = np.array([-2.0, -1.9733333333333334, -1.9466666666666668, -1.92, -1.8933333333333333, -1.8666666666666667, -1.84, -1.8133333333333332, -1.7866666666666666, -1.76, -1.7333333333333334, -1.7066666666666666, -1.68, -1.6533333333333333, -1.6266666666666667, -1.6, -1.5733333333333333, -1.5466666666666666, -1.52, -1.4933333333333332, -1.4666666666666668, -1.44, -1.4133333333333333, -1.3866666666666667, -1.3599999999999999, -1.3333333333333333, -1.3066666666666666, -1.2799999999999998, -1.2533333333333334, -1.2266666666666666, -1.2, -1.1733333333333333, -1.1466666666666665, -1.12, -1.0933333333333333, -1.0666666666666667, -1.04, -1.0133333333333332, -0.9866666666666666, -0.96, -0.9333333333333333, -0.9066666666666665, -0.8799999999999999, -0.8533333333333333, -0.8266666666666667, -0.7999999999999998, -0.7733333333333332, -0.7466666666666666, -0.72, -0.6933333333333334, -0.6666666666666665, -0.6399999999999999, -0.6133333333333333, -0.5866666666666667, -0.5599999999999998, -0.5333333333333332, -0.5066666666666666, -0.48, -0.45333333333333314, -0.4266666666666665, -0.3999999999999999, -0.3733333333333333, -0.34666666666666646, -0.31999999999999984, -0.2933333333333332, -0.2666666666666666, -0.24, -0.21333333333333315, -0.18666666666666654, -0.15999999999999992, -0.1333333333333333, -0.10666666666666647, -0.07999999999999985, -0.05333333333333323, -0.026666666666666616, 0.0, 0.02666666666666684, 0.05333333333333368, 0.08000000000000007, 0.10666666666666691, 0.1333333333333333, 0.16000000000000014, 0.18666666666666698, 0.21333333333333337, 0.2400000000000002, 0.2666666666666666, 0.29333333333333345, 0.3200000000000003, 0.3466666666666667, 0.3733333333333335, 0.40000000000000036, 0.42666666666666675, 0.4533333333333336, 0.48, 0.5066666666666668, 0.5333333333333337, 0.56, 0.5866666666666669, 0.6133333333333333, 0.6400000000000001, 0.666666666666667, 0.6933333333333334, 0.7200000000000002, 0.746666666666667, 0.7733333333333334, 0.8000000000000003, 0.8266666666666667, 0.8533333333333335, 0.8800000000000003, 0.9066666666666667, 0.9333333333333336, 0.9600000000000004, 0.9866666666666668, 1.0133333333333336, 1.04, 1.0666666666666669, 1.0933333333333337, 1.12, 1.146666666666667, 1.1733333333333333, 1.2000000000000002, 1.226666666666667, 1.2533333333333334, 1.2800000000000002, 1.306666666666667, 1.3333333333333335, 1.3600000000000003, 1.3866666666666667, 1.4133333333333336, 1.4400000000000004, 1.4666666666666668, 1.4933333333333336, 1.52, 1.5466666666666669, 1.5733333333333337, 1.6, 1.626666666666667, 1.6533333333333338, 1.6800000000000002, 1.706666666666667, 1.7333333333333334, 1.7600000000000002, 1.786666666666667, 1.8133333333333335, 1.8400000000000003, 1.8666666666666667, 1.8933333333333335, 1.9200000000000004, 1.9466666666666668, 1.9733333333333336, 2.0, 2.0266666666666673, 2.0533333333333337, 2.08, 2.1066666666666674, 2.1333333333333337, 2.16, 2.1866666666666665, 2.213333333333334, 2.24, 2.2666666666666666, 2.293333333333334, 2.3200000000000003, 2.3466666666666667, 2.373333333333334, 2.4000000000000004, 2.4266666666666667, 2.453333333333334, 2.4800000000000004, 2.506666666666667, 2.533333333333333, 2.5600000000000005, 2.586666666666667, 2.6133333333333333, 2.6400000000000006, 2.666666666666667, 2.6933333333333334, 2.7200000000000006, 2.746666666666667, 2.7733333333333334, 2.8000000000000007, 2.826666666666667, 2.8533333333333335, 2.88, 2.906666666666667, 2.9333333333333336, 2.96, 2.9866666666666672, 3.0133333333333336, 3.04, 3.0666666666666673, 3.0933333333333337, 3.12, 3.1466666666666674, 3.173333333333334, 3.2, 3.2266666666666666, 3.253333333333334, 3.2800000000000002, 3.3066666666666666, 3.333333333333334, 3.3600000000000003, 3.3866666666666667, 3.413333333333334, 3.4400000000000004, 3.466666666666667, 3.493333333333334, 3.5200000000000005, 3.546666666666667, 3.5733333333333333, 3.6000000000000005, 3.626666666666667, 3.6533333333333333, 3.6800000000000006, 3.706666666666667, 3.7333333333333334, 3.7600000000000007, 3.786666666666667, 3.8133333333333335, 3.8400000000000007, 3.866666666666667, 3.8933333333333335, 3.920000000000001, 3.946666666666667, 3.9733333333333336, 4.0, 4.026666666666667, 4.053333333333334, 4.08, 4.106666666666667, 4.133333333333334, 4.16, 4.186666666666667, 4.213333333333334, 4.24, 4.2666666666666675, 4.293333333333334, 4.32, 4.346666666666667, 4.373333333333334, 4.4, 4.426666666666667, 4.453333333333334, 4.48, 4.506666666666667, 4.533333333333334, 4.5600000000000005, 4.586666666666667, 4.613333333333334, 4.640000000000001, 4.666666666666667, 4.693333333333333, 4.720000000000001, 4.746666666666667, 4.773333333333333, 4.800000000000001, 4.826666666666667, 4.8533333333333335, 4.880000000000001, 4.906666666666667, 4.933333333333334, 4.960000000000001, 4.986666666666667, 5.013333333333334, 5.04, 5.066666666666667, 5.093333333333334, 5.12, 5.146666666666667, 5.173333333333334, 5.2, 5.2266666666666675, 5.253333333333334, 5.28, 5.3066666666666675, 5.333333333333334, 5.36, 5.386666666666667, 5.413333333333334, 5.44, 5.466666666666667, 5.493333333333334, 5.5200000000000005, 5.546666666666667, 5.573333333333334, 5.6000000000000005, 5.626666666666667, 5.653333333333334, 5.680000000000001, 5.706666666666667, 5.733333333333333, 5.760000000000001, 5.786666666666667, 5.8133333333333335, 5.840000000000001, 5.866666666666667, 5.8933333333333335, 5.920000000000001, 5.946666666666667, 5.973333333333334])
p_top = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 52.872576794481944, 60.36328831067402, 68.58181676019626, 77.54010551102007, 87.24546077586182, 97.71380116903708, 108.97588971792945, 121.1230000000001, 134.28859358309765, 148.68731887019467, 164.5633871885017, 182.18666227169558, 201.83818349095438, 223.81267643614873, 248.42818042617202, 276.03506399135284, 307.0337209116861, 341.87896590372986, 381.1192679598341, 425.3893723626624, 475.5021283556349, 532.4158997232969, 597.4090000000003, 672.0410742621983, 758.533379251738, 859.7165530846363, 979.6593316087284, 1124.0206760913093, 1301.1409742106393, 1519.6640526904762, 1788.227505227972, 2115.468518070886, 2510.024265626171, 2980.5319397027483, 3535.6285264165717, 4183.950889387946, 4934.135917372832, 5794.82045, 4934.420403031789, 4184.487017193709, 3536.3779199568417, 2981.450849155281, 2511.0636810971228, 2116.574298226707, 1789.340587092605, 1520.720465547997, 1302.0718287570105, 1124.752573196128, 980.1205102605351, 859.8428649405212, 758.281365651945, 671.394251505908, 596.3759999999991, 531.0259774866203, 473.7897372355332, 423.3784182587137, 378.7905732211398, 339.2052390738119, 304.05693724550565, 272.8094869362206, 244.93161029019166, 220.00582429999793, 197.7284586919518, 177.8007936067852, 159.92938926933715, 143.86679302868998, 129.3920578431436, 116.3009999999999, 104.39987683440208, 93.51271797907324, 83.48433620575351, 74.20490406269317, 65.59711336145308, 57.63198789171022, 50.29487274165398, 43.58296879663324, 37.48630089022281, 31.9864346061135, 27.059460686269396, 22.676772829659317, 18.800931214057726, 15.390398961980129, 12.405999999999976, 9.808750398065335, 7.556975821856745, 5.609771290609389, 3.930241859207573, 2.4840486871436225, 1.2405555333284073, 0.17249876364223482, -0.7431963631258247, -1.527178008314559, -2.1976614885762182, -2.7685586963324997, -3.2507065483663036, -3.654521247585481, -3.989819778712002, -4.264, -4.483339654609878, -4.653107428963594, -4.778317791591804, -4.863979954625947, -4.914753159149808, -4.934618850520453, -4.92747858548951, -4.8971925826874845, -4.846723689404206, -4.778134576732593, -4.693446251582014, -4.5947146119860145, -4.483931113002393, -4.363044061334697, -4.234, -4.0985235604644705, -3.9574567255996067, -3.811418823406809, -3.661031677747074, -3.507335845802052, -3.3521869245809506, -3.197532929895521, -3.045306521322895, -2.897111875210079, -2.7542247897630006, -2.6179055748822746, -2.4892499037088855, -2.3679261099625046, -2.2528674758007727, -2.142999999999998, -2.03745860943027, -1.936218329497232, -1.8394781167611673, -1.747441997712195, -1.6600861278953287, -1.5769440991709724, -1.4974984903385995, -1.4212457390197035, -1.3480060418851023, -1.277923339237923, -1.2111555085799302, -1.1478331446399963, -1.0878510870420735, -1.030982650405997, -0.9769999999999986, -0.9257060180247605, -0.8770268056960342, -0.8309191877527421, -0.7873394579322428, -0.7462046341344566, -0.707355543280765, -0.6706241363212588, -0.6358440114180867, -0.6028906779941965, -0.5716816224666781, -0.542135986875562, -0.514173025332713, -0.4877142855378665, -0.4626824379381182, -0.43899999999999995, -0.4165943842589561, -0.3954130579480966, -0.37540839142719296, -0.35653259042310337, -0.3387377083468086, -0.3219756460666744, -0.3061981561026408, -0.2913568256017715, -0.27740309673209135, -0.26428826973514485, -0.25196348850713185, -0.24038028798051297, -0.22949466726113874, -0.21926484416543945, -0.20964892669615007, -0.20060488653428893, -0.19209054169683576, -0.18406358448408644, -0.17648157320150093, -0.16930192351620832, -0.1624822410268282, -0.15599398273189863, -0.1498278946257743, -0.143976027663664, -0.1384303016449746, -0.13318251846105061, -0.1282243553559818, -0.12354735885956175, -0.11914296564285694, -0.115, -0.11110306063818942, -0.10743823387446969, -0.10399057427215586, -0.10074506770538746, -0.09768595719983417, -0.09478686404733155, -0.09203198506265693, -0.08941054762959841, -0.0869117305275437, -0.08452465786411881, -0.08223840650397521, -0.08004200111956421, -0.07792441608596867, -0.07587457925344288, -0.07388137031451006, -0.0719336220194075, -0.07002011928032632, -0.06812960289914699, -0.06625080975713306, -0.06437774354263374, -0.062514670658265, -0.06066699908536895, -0.058840093159610946, -0.05703927761570363, -0.055269833348506214, -0.053537002547862714, -0.05184598638412147, -0.05020194624570422, -0.048609299571983254, -0.0470683062510954, -0.04557769671809728, -0.04413616149903207, -0.04274235499024258, -0.04139489568258407, -0.04009236691353722, -0.03883331623326931, -0.037616257023266514, -0.03643968154837971, -0.03530235422607354, -0.03420331195177105, -0.03314152488588915, -0.032107966203906546, -0.031098116639678777, -0.03011436052645273, -0.029159083111374974, -0.02823466944709844, -0.027343502994046268, -0.026487055881413483, -0.025664282278113962, -0.024873704934188466, -0.024113846492361448, -0.023383229629240988, -0.022680377059935815, -0.022003811622373776, -0.02135205612181401, -0.020723633222147094, -0.020117107909086978, -0.01953141316798061, -0.01896567156149227, -0.018419007242562745, -0.017890544187225972, -0.017379406521364806, -0.016884718251067944, -0.016405603478766607, -0.015941186322196526, -0.015490596420519654, -0.01505321655270402, -0.014628780797528938, -0.014217049100448345, -0.01381778138256858, -0.01343073756640109, -0.013055677585210439, -0.012692361374500867, -0.012340548869974452])
def leps(z, D, alpha, r_eq):
return (D*(np.exp(-2*alpha*(z - r_eq)) - 2*np.exp(-alpha*(z - r_eq))))
popt, pcov = curve_fit(leps, z_top, p_top)
print(popt)
plt.figure(figsize=(20,10))
plt.plot(z_top,p_top,label='data')
plt.plot(z_top, leps(z_top, *popt), 'r-',label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
plt.ylim(-8,1)
EDIT:
the fit is very poor, even after providing starting parameters. How do you determine the optimal parameters for a good fit?

How to choose the mesh for phonon calculation with PyIron

I would like to calculate phonon density of states and band structure with pyiron, using the phononpy package.
I created a job, following the tutorial:
phono = pr.create_job(pr.job_type.PhonopyJob,"pDOS")
I can run this job, but it takes a lot of time because the mesh is too dense. Is there a way of choosing the mesh I would like to work with ?
Also, I would like to calculate phonon band structure for a given path, is it possible with pyiron ?

You can specify the input in:
phono.input
Here you can set the mesh as:
phono.input["dos_mesh"]
Best,
Jan
To address the comment regarding the band structure - you can use the phonopy API directly:
bands = []
q_start = np.array([0.5, 0.5, 0.0])
q_end = np.array([0.0, 0.0, 0.0])
band = []
for i in range(51):
band.append(q_start + (q_end - q_start) / 50 * i)
bands.append(band)
q_start = np.array([0.0, 0.0, 0.0])
q_end = np.array([0.5, 0.0, 0.0])
band = []
for i in range(51):
band.append(q_start + (q_end - q_start) / 50 * i)
bands.append(band)
phon.phonopy.set_band_structure(bands)
phon.phonopy.plot_band_structure().show()

Is it possible to alias multiple names in a numpy record array?

Suppose I construct a numpy record array like this
num_rows = <whatever>
data = np.zeros(
(num_rows,),
dtype={
'names':['apple', 'banana'],
'formats': ['f8', 'f8']
}
Now I can access data either by name or index.
For example, the following are the same:
data['banana'][0]
and
data[0]['banana']
etc.
Is there a way to alias different names?
For example, can I set things up so that there's another name manzana such that
data['manzana']
is the same thing as
data['apple']
?

['offsets' and 'titles' are 2 mechanisms for giving different names to fields]
There is an offset parameter that can function in this way. Usually it is used to split another field into several pieces (e.g. an int into bytes). But it also works with identical fields. In effect it defines several fields with overlapping data.
In [743]: dt=np.dtype({'names':['apple','manzana','banana','guineo'],
'formats':['f8','f8','f8','f8'],
'offsets':[0,0,8,8]})
In [745]: np.zeros((3,),dtype=dt)
Out[745]:
array([(0.0, 0.0, 0.0, 0.0), (0.0, 0.0, 0.0, 0.0), (0.0, 0.0, 0.0, 0.0)],
dtype={'names':['apple','manzana','banana','guineo'],
'formats':['<f8','<f8','<f8','<f8'],
'offsets':[0,0,8,8], 'itemsize':16})
In [746]: A=np.zeros((3,),dtype=dt)
In [747]: A['banana']=[1,2,3]
In [748]: A
Out[748]:
array([(0.0, 0.0, 1.0, 1.0),
(0.0, 0.0, 2.0, 2.0),
(0.0, 0.0, 3.0, 3.0)],
dtype={'names':['apple','manzana','banana','guineo'], 'formats':['<f8','<f8','<f8','<f8'], 'offsets':[0,0,8,8], 'itemsize':16})
In [749]: A['guineo']
Out[749]: array([ 1., 2., 3.])
In [750]: A['manzana']=[.1,.2,.3]
In [751]: A['apple']
Out[751]: array([ 0.1, 0.2, 0.3])
In [752]: A
Out[752]:
array([(0.1, 0.1, 1.0, 1.0),
(0.2, 0.2, 2.0, 2.0),
(0.3, 0.3, 3.0, 3.0)],
dtype={'names':['apple','manzana','banana','guineo'], 'formats':['<f8','<f8','<f8','<f8'], 'offsets':[0,0,8,8], 'itemsize':16})
There's another dtype parameter, titles that is better suited to your needs, and easier to understand:
http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
In [792]: dt1=np.dtype({'names':['apple','banana'],'formats':['f8','f8'], 'titles':['manzana', 'guineo'], 'offsets':[0,8]})
In [793]: A1=np.zeros((3,),dtype=dt1)
In [794]: A1
Out[794]:
array([(0.0, 0.0), (0.0, 0.0), (0.0, 0.0)],
dtype=[(('manzana', 'apple'), '<f8'), (('guineo', 'banana'), '<f8')])
In [795]: A1['apple']=[1,2,3]
In [796]: A1['guineo']=[.1,.2,.3]
In [797]: A1
Out[797]:
array([(1.0, 0.1), (2.0, 0.2), (3.0, 0.3)],
dtype=[(('manzana', 'apple'), '<f8'), (('guineo', 'banana'), '<f8')])
In [798]: A1['banana']
Out[798]: array([ 0.1, 0.2, 0.3])

I have put the answer of #hpaulj into a simple method and share it here in case someone wants to use it.
def add_alias(arr, original, alias):
"""
Adds an alias to the field with the name original to the array arr.
Only one alias per field is allowed.
"""
if arr.dtype.names is None:
raise TypeError("arr must be a structured array. Use add_name instead.")
descr = arr.dtype.descr
try:
index = arr.dtype.names.index(original)
except ValueError:
raise ValueError("arr does not have a field named '" + str(original)
+ "'")
if type(descr[index][0]) is tuple:
raise ValueError("The field " + str(original) +
" already has an alias.")
descr[index] = ((alias, descr[index][0]), descr[index][1])
arr.dtype = np.dtype(descr)
return arr
def add_name(arr, name):
"""
Adds a name to the data of an unstructured array.
"""
if arr.dtype.names is not None:
raise TypeError("arr must not be a structured array. "
+ "Use add_alias instead.")
arr.dtype = np.dtype([(name, arr.dtype.name)])
return arr

removing baseline signal using fourier transforms

I have timeseries data for many terms an example of which is below:
term1 = [0.0, 0.0, 0.0, 0.0, 2.2384935833581433e-06, 3.938767914008819e-06, 0.0, 0.0, 1.1961851263949013e-06, 0.0, 2.278384397623645e-06, 1.100158422812885e-06, 0.0, 1.095521835393462e-06, 0.0, 0.0, 1.6933152148605343e-06, 0.0, 8.460737945563612e-07, 8.949410770794851e-07, 0.0, 2.8698467119209605e-06, 0.0, 0.0, 0.0, 3.9163008188985015e-06, 2.2244961516216576e-06, 0.0, 0.0, 1.9407903674692482e-06, 0.0, 0.0, 0.0, 0.0, 9.514657329616274e-07, 1.94463053478312e-06, 0.0, 0.0, 0.0, 2.0373216961518047e-06, 1.8835690620014428e-06, 0.0, 0.0, 0.0, 0.0, 9.707946148081127e-07, 0.0, 0.0, 1.6121985390256838e-06, 1.9547361301697883e-06, 0.0, 2.2876018840689116e-06, 2.208826914114183e-06, 1.9640500282823203e-06, 0.0, 2.6234669115235785e-06, 0.0, 0.0, 0.0, 1.986207773222741e-06, 1.049193537387487e-06, 1.090723073046815e-06, 0.0, 1.0257546476943088e-06, 9.179053033814713e-07, 0.0, 0.0, 0.0, 0.0, 9.335621182897889e-07, 0.0, 0.0, 0.0, 0.0, 2.1267500494469387e-06, 2.215050381320923e-06, 2.163720040591388e-06, 1.937729136470388e-06, 1.6037643556956889e-06, 1.313906783569333e-06, 0.0, 1.0064645216223805e-06, 1.876346865234201e-06, 9.504447606257348e-07, 2.017974095266539e-06, 0.0, 2.120782823355757e-06, 0.0, 0.0, 0.0, 0.0, 9.216394491176685e-07, 0.0, 0.0, 1.0401357169083422e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0089962853658684e-06, 1.8249773702806084e-06, 0.0, 1.2890950295073852e-06, 5.42812725267281e-06, 1.9185480428411778e-06, 2.6955316172381044e-06, 0.0, 0.0, 1.0070239923466176e-06, 0.0, 1.021152145542773e-06, 9.919749228739498e-07, 1.9293082175989564e-06, 9.802489636317832e-07, 1.0483850676418046e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 1.9369409504181854e-06, 0.0, 4.619620451983665e-06, 0.0, 6.0795324434248845e-06, 0.0, 1.5312669396405198e-06, 1.2797051559320733e-06, 1.1002903666277531e-06, 0.0, 1.0054768323055684e-06, 2.060260561153169e-06, 1.0898719291496056e-06, 3.4605907920600203e-06, 3.3500051925080486e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 6.5521496510980315e-06, 0.0, 0.0, 0.0, 3.01862187836765e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.849817053093449e-06, 6.5552277941658475e-06, 1.985771944021089e-06, 1.010233667047188e-06, 9.802307070992228e-07, 5.605931075077432e-06, 3.651067480854715e-06, 0.0, 0.0, 2.9476960807432912e-06, 1.834478659509754e-06, 0.0, 0.0, 0.0, 0.0, 3.3801712394749917e-06, 0.0, 2.2884970981856794e-06, 1.02014792144861e-06, 2.906143199237428e-06, 9.807873564740302e-07, 0.0, 2.106593638087213e-06, 3.0329622335542676e-06, 2.9093758515985565e-06, 0.0, 2.12762335960239e-06, 9.614820669172289e-07, 9.264114341404848e-07, 0.0, 0.0, 9.073611487918033e-07, 0.0, 0.0, 0.0, 6.0360958532021484e-06, 0.0, 4.553288270957079e-06, 2.0712553257152562e-06, 3.292603824030081e-06, 2.690786880261329e-06, 2.301011409565074e-06, 2.029661472762958e-06, 0.0, 9.657114492818003e-07, 9.948942029504583e-07, 1.028682761437152e-06, 2.0694207898151387e-06, 3.845369982272845e-06, 9.048250701691842e-07, 1.7726379156614332e-06, 0.0, 9.238711680133629e-07, 9.231112912203808e-07, 9.422814896339613e-07, 0.0, 1.2123519263665934e-06, 0.0, 0.0, 2.1675188628329036e-06, 0.0, 4.498718989767663e-06, 0.0, 0.0, 2.650273839544471e-06, 1.1954029583832415e-06, 4.180999656112778e-06, 1.9036523473937095e-06, 9.75877289286136e-07, 0.0, 2.093618232902467e-06, 1.032899928523325e-06, 0.0, 4.473312219299659e-06, 8.762705923589204e-07, 0.0, 0.0, 1.792797436299666e-06, 0.0, 0.0, 1.1974513445582422e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.1404264054329915e-06, 3.061324451410658e-06, 9.84829683554526e-07, 2.932895354293759e-06, 2.0897069394988045e-06, 0.0, 2.128187093183736e-06, 0.0, 4.686861415132188e-06, 6.37755683086446e-06, 1.8420463661490824e-06, 2.8347403094402523e-06, 1.9033842171380715e-06, 6.909144746582441e-06, 0.0, 0.0, 1.5479612576256442e-06, 5.621978186724636e-06, 2.087185930697078e-06, 1.3168406359462377e-05, 1.9676130885622652e-05, 1.9988766313331908e-05, 3.1801079228204546e-05, 3.322824899588385e-05, 2.0358501231090545e-05, 1.2383952049337664e-05, 1.8052256532066507e-05, 7.770543617518302e-06, 9.226179797741636e-06, 4.400430362089412e-06, 4.333084180992927e-06, 7.477274426653279e-06, 3.0526428255261993e-06, 4.952368123389242e-06, 1.2578584707962998e-05, 0.0, 2.121750274236223e-06, 0.0, 2.38940918273843e-06, 0.0, 1.5693511988273807e-06, 0.0, 0.0, 4.520448247648237e-06, 4.0303440122522456e-05, 2.8660979509446863e-05, 2.4793768971660722e-05, 3.957070185234852e-05, 2.64488881248099e-05, 6.428381095168035e-05, 5.6557662521419976e-05, 6.855540059858658e-05, 7.079288025889968e-05, 7.135683422742382e-05, 5.5663480860112103e-05, 8.088436527379357e-05, 7.142268494354861e-05, 8.243171356847987e-05, 7.658173644233611e-05, 5.4275733753644613e-05, 2.7329513031804995e-05, 1.8666856995404658e-05, 2.5061514626811264e-05, 9.707359513272993e-06, 2.233654188450612e-05, 2.0577084330035857e-05, 6.037067595033506e-05, 5.358585847760433e-05, 6.353114888415205e-05, 4.913406130358561e-05, 6.253876100291326e-05, 5.783647108547192e-05, 5.29265883017118e-05, 4.295770587763158e-05, 0.00012513639867455526, 0.0001264425725280477, 0.00010075697417828198, 7.700585441944497e-05, 6.390017630639553e-05, 6.862379380485504e-05, 8.118867124374998e-05, 8.928305705187346e-05, 8.923668314113125e-05, 5.0862818355003976e-05, 2.5192448399293734e-05, 1.9491995287268695e-05, 1.1397180337584482e-05, 1.8548131739430545e-05, 2.8274146120787152e-05, 2.9861740143137274e-05, 5.749201435920551e-05, 8.676081065218611e-05, 0.00011692016691003383, 6.18107213073443e-05, 8.31986307882476e-05, 5.661490072734421e-05, 6.637785526376392e-05, 6.189842468509176e-05, 5.077848495281155e-05, 3.7630726455798414e-05, 6.325167842846687e-05, 7.447442335517917e-05, 7.881778491014126e-05, 8.347575938861497e-05, 6.553610066345062e-05, 6.209221186256924e-05, 4.671174184109858e-05, 4.583301504850661e-05, 2.9423292949863758e-05, 1.9969520206001368e-05, 1.3386836054765546e-05, 1.0233804045678584e-05, 2.3371876153986385e-05, 3.701784260013326e-05, 2.6804842191646374e-05, 3.729558727386808e-05, 7.011179438698544e-05, 4.616049584765358e-05, 6.019787395273405e-05, 8.312188292939014e-05, 6.281596430043117e-05, 6.370630077282333e-05, 6.169767733530766e-05, 6.099512039036877e-05, 7.192322709245217e-05, 6.727547574464268e-05, 4.891125624919348e-05, 8.775231227342841e-05, 9.349358010749929e-05, 4.85363097385816e-05, 4.475820776946539e-05, 1.9528637281926147e-05, 1.5243002033035396e-05, 1.4322461630125293e-05, 1.0492122514416176e-05, 1.1956759574674148e-05, 1.5232250274180506e-05, 3.394641638997643e-05, 2.6115894879792267e-05, 4.868559048521277e-05, 5.612535494090208e-05, 3.269545148571978e-05, 4.967751016319062e-05, 4.8382804751191425e-05, 5.1860846075881435e-05, 4.4034258653232213e-05, 5.362193446127224e-05, 6.213052893181175e-05, 8.561827093901839e-05, 5.877682625663455e-05, 0.0, 0.0, 7.105805443046969e-05, 0.0, 0.0, 2.31393994554528e-05, 7.05044594070575e-06, 2.21491300929156e-05, 4.926848615186025e-06, 1.0752514744385843e-05, 1.4745260873155369e-05, 1.976297604068538e-05, 3.094705732168692e-05, 5.068338091939653e-05, 2.655137469742496e-05, 3.0142790705685793e-05, 3.89279249469607e-05, 6.264176821226988e-05, 3.598536226187379e-05, 4.430195278344506e-05, 2.7501831818440764e-05, 1.7243328268903956e-05, 1.2049184772240285e-05, 2.1016880758625327e-05, 3.411070201956675e-05, 3.1893789428697184e-05, 1.8509911029027654e-05, 3.920735117199027e-05, 3.700840501998454e-05, 8.529330234343347e-06, 1.1007881643256571e-05, 4.661265813344272e-06, 7.306007242688513e-06, 2.6772256446090046e-06, 3.0075821145106816e-06, 6.713527085725027e-06, 2.204123915846549e-05, 7.880065404542858e-06, 4.3539870647002475e-05, 6.0898558226633984e-05, 7.956054903697144e-05, 4.80968670903199e-05, 3.476307626484116e-05, 3.233622280581405e-05, 4.097520999795124e-05, 1.6048981491512094e-05, 3.4725910431663494e-05, 2.3840743831207534e-05, 4.194630872483221e-05, 3.472531193096608e-05, 2.9240209403218155e-05, 2.5871727972711297e-05, 1.1918039641386187e-05, 1.2189485552920143e-05, 8.254477280067191e-06, 5.343416003103456e-06, 0.0, 4.795714549478586e-06, 6.705621859254362e-06, 9.484831383410081e-06, 2.503719812292549e-05, 1.9037212038371403e-05, 2.448114104715256e-05, 3.2063674685728836e-05, 2.73499598297465e-05, 2.6255716088190032e-05, 2.930473870366029e-05, 2.490020970307041e-05, 2.4037259675477766e-05, 1.8683888229243836e-05, 9.573344744760269e-06, 2.01589736663327e-05, 2.8955116521484698e-05, 1.934869527601605e-05, 2.1111566182648825e-05, 1.0035410663340645e-05, 4.154485944681635e-06, 8.468739061212046e-06, 8.415088253238056e-06, 1.3883239181832948e-06, 0.0, 2.9995080806747692e-06, 1.6303266848611124e-06, 3.714448088730736e-06, 8.976418947425114e-06, 9.729566693747293e-06, 3.3588780313874236e-05, 1.7154466165266127e-05, 1.9646193877372823e-05, 9.475852684603824e-06, 9.763432041631274e-06, 2.5840349706066022e-05, 1.4272109443725072e-05, 2.262309793162043e-05, 1.733067926359468e-05, 8.405046389852468e-06, 1.6489619195801272e-05, 6.6721749177376435e-06, 5.2645543870584616e-06, 5.563468043439559e-06, 5.668953522517651e-06, 2.564151874715539e-06, 4.72535638047152e-06, 1.1322053548784465e-06, 4.683593955822e-06, 5.170243182388084e-06, 1.4458242427134072e-06, 5.110793484760465e-06, 8.06295555698897e-06, 1.7613618850094893e-05, 1.3702227753862316e-05, 1.2582942563061514e-05, 1.5863866870429223e-05, 5.763738591399926e-06, 5.010013765012819e-06, 3.355941190486578e-06, 1.2264709219075303e-05, 3.0533139142568385e-06, 5.2266756983622735e-06, 3.0845411025383717e-06, 7.013177761012944e-06, 1.5042033081191253e-05, 7.918060391926394e-06, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0146814988996414e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 1.4253135689851767e-05, 0.0, 1.4218885523752648e-05, 0.0, 1.539361472861057e-05, 0.0, 3.981789947307646e-05, 0.0, 1.2433017120264575e-05, 1.2777756481516975e-05, 1.2764382267720154e-05, 0.0, 1.2131652695046646e-05, 0.0, 0.0, 1.9324418335008117e-05, 0.0, 5.3279343598486864e-05, 1.316118503310038e-05, 8.202637968370628e-06, 8.606938339893733e-06, 2.2898281255009e-06, 0.0, 3.510274573677153e-06, 1.6317872149471709e-06, 4.578600840631114e-06, 3.877291479264245e-06, 2.8741881616021876e-06, 0.0, 1.0729671296519832e-05, 4.808871405969733e-06, 4.534612698729401e-06, 4.3333188889370365e-06, 0.0, 2.743032696949748e-06, 4.019804235533729e-06, 1.8078426019917002e-06, 5.444991968636846e-06]
Each element is the combined signal for an hour and the list is over 24 days. Therefore, should have 24 days * 24 hours = 576 elements.
I am interested in the dynamics of the signal of each term over the course of days. However, confounding to this is the baseline changes of the signal within a day. I also have time series for basic terms that capture this baseline signal during a day such as the following.
baseline = [0.0056738537419516195, 0.005420397434626666, 0.005019676698052322, 0.004214006968007205, 0.004143451622795924, 0.00373395198248036, 0.0037080495988714344, 0.0036409523281401525, 0.003919898659196092, 0.004388163261294729, 0.004595501330006892, 0.005097033972892097, 0.0052221285817481335, 0.005184009325081863, 0.005273633551787361, 0.005053393415305126, 0.004444952439008902, 0.004552838940992971, 0.004808237374463801, 0.004895327691624783, 0.005059086256629757, 0.005598114319387153, 0.005952632334681949, 0.005717805004263755, 0.006126432469142252, 0.005592477569387059, 0.004920585487387107, 0.004318038669070883, 0.003877225571288378, 0.0036583898426795327, 0.0037336953886437474, 0.0037760782770061294, 0.0042338376814351954, 0.004192003050341723, 0.0046450557083186645, 0.004900468947653463, 0.005272546953959605, 0.005265723105151999, 0.0052304537716869855, 0.005121826744125637, 0.005224078793461002, 0.00501027352884918, 0.004871995260153345, 0.004863044486978714, 0.005310347911635811, 0.0058606870895965765, 0.00596470801561322, 0.005997180909289017, 0.00588291246890472, 0.005328610690842843, 0.004941976393965633, 0.004426509645673344, 0.0041172533679088375, 0.0038888190559989945, 0.003785501144341545, 0.0038683019610415165, 0.003826474222198437, 0.004178336738982966, 0.004574137078717032, 0.004854291797756379, 0.005216590267890586, 0.00514712218170792, 0.005217487414098377, 0.005239554740422529, 0.005138433888329476, 0.0050591314342241745, 0.005099277335119803, 0.00469742744667216, 0.005140145739820509, 0.005534156237221868, 0.006098190503066302, 0.00627542293362276, 0.005859315099582288, 0.0055863100804189264, 0.005193523620749424, 0.004680401455731111, 0.0041370327176107335, 0.003790198190936078, 0.0037143182477912154, 0.0037406926128218908, 0.0038838040372017974, 0.00413455625482474, 0.004309030576010342, 0.004768381364059312, 0.004905695025592956, 0.0050965947056771715, 0.005178951654634759, 0.005250840996574289, 0.005083679897873849, 0.0050438189257025106, 0.00465730975130931, 0.004511425103430987, 0.004631293617276186, 0.0049417738509291015, 0.005495036992426772, 0.0056409591251836465, 0.005487421237451456, 0.005093572532544252, 0.005043698439924855, 0.004837771685295603, 0.0038251289273366134, 0.003817852658627033, 0.003792612420533331, 0.003922716174790973, 0.00412646233748187, 0.004534488299255124, 0.004712687777471286, 0.005096266809417225, 0.00523394116440575, 0.005257672264041691, 0.005305086574696615, 0.005100654966986151, 0.004965826463906992, 0.005073115958176456, 0.00469441228683261, 0.004553136348768357, 0.004723653823501124, 0.004726168081415059, 0.005290955031631742, 0.005325956759690025, 0.005453676000994151, 0.005531903354394338, 0.005121462750913455, 0.004790546408707061, 0.004460326025284444, 0.003982093750443299, 0.0036151869988949132, 0.0035295702958713982, 0.003722662298606401, 0.004089779292755358, 0.004116707488056058, 0.004463311658604419, 0.004863245054602056, 0.005019950105663084, 0.005111292599872651, 0.0050328244675445916, 0.004886511461492081, 0.005017059119637564, 0.004997550003214928, 0.004989853142609061, 0.004888243576205561, 0.004801721031771264, 0.005142349216675533, 0.0053550501391269115, 0.00510410900976245, 0.005113311675603742, 0.004865951202283446, 0.004739388247627576, 0.004314592960862043, 0.003932197365205607, 0.0036889365827877003, 0.003444247563217489, 0.0033695476656641706, 0.003779678994400599, 0.004182362477080399, 0.004650999598571368, 0.004964528816231351, 0.005246502668776329, 0.005150211093436487, 0.0051813375657147505, 0.005326590813316477, 0.00501407415865325, 0.004920848192186853, 0.005020741681762219, 0.005108871853087233, 0.004991922013198609, 0.005551866678436957, 0.005681472655730911, 0.005624204122058199, 0.005202581478369662, 0.00490495583623749, 0.0043628317352519584, 0.0037568042368143423, 0.0035018559432594323, 0.0035627004864066413, 0.003560172130774401, 0.003604382929642445, 0.003782708492731446, 0.003958167037361377, 0.004405696805281344, 0.004888234197579893, 0.004849378554876764, 0.005035728295111269, 0.005150565049279978, 0.005104177573029002, 0.005540331228404623, 0.005146813504207926, 0.004991504807148932, 0.0050371760815936415, 0.005174258383207836, 0.005598418288045426, 0.0056576481335463774, 0.00561832393839059, 0.005408391628077189, 0.0052292710408241285, 0.004705309149638305, 0.003924934489565002, 0.003854606161156092, 0.0038935040219155712, 0.003830335124052002, 0.003746046574771941, 0.003865490274877053, 0.004168222873979538, 0.0045871293840885514, 0.004915772256778214, 0.005072434696646597, 0.00492522147976003, 0.004978792784547765, 0.004963870334948144, 0.004955409293231536, 0.004709890770618299, 0.004888202349958703, 0.0051805005663287775, 0.005568883603736712, 0.005781789868618008, 0.006061759631832967, 0.005730308168750368, 0.0055273545529884146, 0.005050318950400666, 0.004505314632141857, 0.0041320733921015994, 0.0037557073980650723, 0.0034979193552635043, 0.0037461620721961097, 0.0036352203964434373, 0.003974040173135196, 0.004094756199243869, 0.004649079406159152, 0.004920019940715673, 0.005231951964023264, 0.005121117845618645, 0.005064423379922766, 0.00498326981229982, 0.004871188222923238, 0.004660839287914527, 0.0047034466283560495, 0.004866548640835444, 0.005578880008506938, 0.0059683185805929845, 0.006061498706153822, 0.005800490254423062, 0.0054633509277901724, 0.004921961696040911, 0.004376719066835311, 0.00393610914724284, 0.0037954515471031775, 0.003581690980473693, 0.003563708289302751, 0.0037463007418473766, 0.00403278474399164, 0.004356886520045223, 0.004787462849992179, 0.005179338649547787, 0.005143654461390953, 0.005203417442834235, 0.005153892139635152, 0.005114303176192244, 0.00504646961230832, 0.00478839952880454, 0.004711338394289699, 0.004911682972324793, 0.005442432018950797, 0.005865476365139558, 0.006157467255298909, 0.005776991413458904, 0.00537648513923766, 0.005215877640811999, 0.004586994881879395, 0.00404235177861292, 0.0038098588593210615, 0.003611933103919232, 0.003782482344031445, 0.003847756732676113, 0.004015496451997738, 0.004222327790973872, 0.004767228509347478, 0.005026217727591916, 0.005032992226639765, 0.0051856184936032845, 0.005070660243331873, 0.005025667638424633, 0.004771111450073196, 0.0049169687623427365, 0.0, 0.004725137860724068, 0.00480564403797717, 0.004993865191923319, 0.005382243541508231, 0.005436232552738047, 0.005416886729676188, 0.004777014387860352, 0.0048255785043644925, 0.004081842852408802, 0.004090331218562488, 0.00378104976817826, 0.003521792464859018, 0.0036283065618489215, 0.003818665737661915, 0.003988803567300145, 0.004483523199147563, 0.004696601941747573, 0.005206918843848881, 0.005231253931233336, 0.005154439277447777, 0.005107271378732522, 0.004862372011026066, 0.005097539245443387, 0.004771922511620435, 0.004800155668906229, 0.004886324331150043, 0.005186594367994167, 0.0055550364814704704, 0.00565254113064783, 0.00542892074907446, 0.005216026402108949, 0.0050842262523550985, 0.004506330112231061, 0.004262871158699087, 0.004073705404217544, 0.003562133424289835, 0.003499455612234611, 0.0037587992642927636, 0.004170545895025578, 0.004646029409170125, 0.004941082109950799, 0.005336110809450001, 0.005238846272634943, 0.0051019151317224926, 0.004828998520466023, 0.00470819320853546, 0.004974373055097931, 0.004975308413634935, 0.005266317039295838, 0.005489162450620279, 0.005606273008057806, 0.00603476714807901, 0.0061970275556501725, 0.0058349840239690235, 0.005192678736923442, 0.004639151581343363, 0.004229911816211891, 0.003727661961919841, 0.00375780482393585, 0.0033937487713780225, 0.003400171769633621, 0.003719857252709842, 0.0037474521895174925, 0.004410321140619574, 0.00505109832021614, 0.00506160098731807, 0.005046922423918226, 0.005300710721177051, 0.005104647840739084, 0.004974276083656935, 0.004902745159619985, 0.005039594632444682, 0.005189007878086687, 0.005840559146565768, 0.005924790523904985, 0.006041782063467494, 0.006054874048959406, 0.005728511142370623, 0.005014567400775691, 0.004479858189014036, 0.004064222403658478, 0.0038690888337760544, 0.0038101713160671666, 0.0038192317788082945, 0.003855643888760465, 0.004151893395194348, 0.004439198456142054, 0.004868610511159107, 0.005164087705238066, 0.0052260812748906515, 0.005049306708959293, 0.005295364532855441, 0.004976241631407976, 0.005325257379808529, 0.004981215539676753, 0.004904617253355752, 0.005133080934624669, 0.005474999665809228, 0.006018281474269119, 0.0059556619441451936, 0.00582564335486158, 0.0057773567703702745, 0.005185870554701607, 0.004927387470357575, 0.004290471577704514, 0.003894605250504856, 0.0036579206650162693, 0.0037227880322444513, 0.0037587839308041025, 0.004025131552347727, 0.0043915455477435165, 0.004973183367291931, 0.005602412946227073, 0.005438255876982902, 0.005057281453194344, 0.0055819722968782305, 0.0052582960278547575, 0.0060302188495155494, 0.003969113083640037, 0.004874700151948723, 0.0048366059153241445, 0.005174590517957408, 0.005237240077942745, 0.005935388138900985, 0.006375850801552381, 0.006218749794135666, 0.005833520305137985, 0.005325978611613252, 0.00473056992525788, 0.0039874605990664344, 0.0038460789847597175, 0.003587065463944717, 0.00384212944765237, 0.004264645875837948, 0.004969973892903938, 0.005856983835337711, 0.006181231788159266, 0.006313470979891048, 0.006097287985997557, 0.005694104539336737, 0.005355534257732001, 0.005274420505031954, 0.004712403572544698, 0.004584515000959549, 0.004766412751530095, 0.0048104263193712886, 0.005309031929686986, 0.006042498279882524, 0.006496377367343072, 0.005619222170751848, 0.005418471293122766, 0.005015661629991529, 0.005062499505228742, 0.004308572994534354, 0.0038880894398937347, 0.003538125785331658, 0.0034843298748529253, 0.003774099147478583, 0.003896742470805163, 0.004541861762097922, 0.004553179667775172, 0.004948038015149709, 0.0050269339456022605, 0.00522398911361471, 0.005050975431726277, 0.005007174429180125, 0.004833758244552214, 0.004670604547693902, 0.00477521510651887, 0.004939453753268834, 0.005239435739336397, 0.005820798534429634, 0.006094069145690364, 0.005673509972509797, 0.005375844111251002, 0.005187640280456208, 0.00476628984541101, 0.004247493846603608, 0.003794806926377327, 0.003435122854871529, 0.003587919312587277, 0.003811897320196127, 0.0042459415490763925, 0.00460744683733153, 0.004807733730818607, 0.005155657515164588, 0.005405463068510853, 0.005224147724524333, 0.005351078308428722, 0.005384714635929638, 0.005362056525935763, 0.0051377016971353075, 0.004941059319359612, 0.004966034655341646, 0.005026256144832193, 0.005442607412384369, 0.0059898202401797275, 0.005612531062072142, 0.005603529527930128, 0.0051493731726657554, 0.004544820351700367, 0.004496920773323335, 0.00424357787751253, 0.0036690501594786006, 0.003700340743778253, 0.0038846659058119253, 0.004159671170598417, 0.004794839922729552, 0.005004852590193807, 0.005163099925195087, 0.005645338914676821, 0.005432262412191398, 0.0050802949835114155, 0.005169574505964038, 0.0052347116826927985, 0.0052757424822272225, 0.0056125420409050475, 0.005578375783486106, 0.005944651628427074, 0.006010407699526147, 0.0061534279769882615, 0.005756457061668538, 0.005283251628717022, 0.004694029423550289, 0.0042271372620665245, 0.003995084263772031, 0.003916612465121526, 0.00385882298694225, 0.0039353658124175695, 0.00403048536977438, 0.0039523025458470164, 0.004692943486212761, 0.005099144811322234, 0.005182029052264465, 0.005496327599559573, 0.0053953892408097875, 0.005256712315751134, 0.004628655585945719, 0.005255300089578979, 0.004727544165215228, 0.005365188522646431, 0.006321448616075385, 0.005962859901186893, 0.0064913517773445605, 0.006403310018717368, 0.005985231247570929, 0.005536676822123271, 0.005652983876148263, 0.0053962798830303575, 0.0036360246130896887, 0.0034235996705107084, 0.004421584551524996, 0.003810299791511898, 0.0038131330853627154, 0.0038483466362599773, 0.005120205311426739, 0.0048344210780759, 0.005090949889906456, 0.005557094917028417, 0.005276073619631902, 0.0056143238257037814, 0.005700457782933553, 0.00584351804652435, 0.004893880732421001, 0.005475919992851946, 0.005248580353868141, 0.005350058838515571, 0.006083169087767963, 0.005703392826945841, 0.006319084795547654, 0.005231157508317081, 0.005381213703447174, 0.005027572682644346, 0.0042202572347266884, 0.004068212855323105, 0.003991170422748069, 0.0037477607718658665, 0.004077183917326014, 0.00408925876065761, 0.004650332253801002, 0.004960348232472058, 0.005144796809267916, 0.00597460791635549, 0.005407754333445995, 0.005265714189536858, 0.005391654498789258, 0.00495731680894397, 0.005033086804203971, 0.00511026991441738, 0.005391897414595909, 0.006005653123816428, 0.0066265552258310415]
i think that a good way for me to extract out the signal I am interested in would be to do a spectral analysis on the timeseries for my terms. The high frequencies should be the daily patterns which I want to get rid of and the lower frequencies should be what I'm interested in. I want to somehow 'divide' my observed signal for a term by the baseline daily signal.
This is my baseline's original signal
and this is my term's original signal
and what i'm trying to do is get something like this in a general way without introducing artefacts. i.e remove the ups and downs that happen every day anyway and capture the general trend.
The naive way I thought of doing this is to first generate ffts for both using numpy(below).
baselines fft
term1s fft
and then create a filter like below
fft2 = fft(term1, n=t)
mgft2=abs(fft2)
plot(mgft2[0:t/2+1])
bp = fft2[:]
for i in range(len(bp)):
if i>=22:
bp[i] = 0
ibp = ifft(bp)
but from what i understand that introduces artefacts, changes the magnitudes and I am not sure how to pick a cutting point. I was hoping for some guidance with respect to implementation in numpy on a better way to divide out my baseline frequencies from my term's frequencies.
thanks

Multiplying (or dividing) in the frequency domain is equivalent to convolving in the time domain. In other words a high-pass FIR filter would remove your low-frequency components directly without going into the frequency domain. If you do go to the frequency domain first be aware that simply removing some frequency components and converting back to time will introduce artifacts. FIR filter design is actually based on choosing filters you can multiply by which also meet your desired frequency specs.
All that said, it sounds like you already know the baselines and you could apply your adjustments directly from your known baselines to your data. The point of filtering would be that it would work without knowing the baselines.

It's one thing to give the code, another to give an explanation. If you just want to look at the spectra, try the following:
import numpy as np
import matplotlib.pyplot as plt
hh = np.hanning(len(term1)) # Use a Hann window to deal with spectral leakage
St = np.fft.rfft(hh*term1)
Sb = np.fft.rfft(hh*baseline)
FSample = 1.0/24 # Sampling frequency is 1/24th of a day
deltaF = 1.0/(len(term1)*FSample) # Frequency resolution is 1/capT = 1/(NumSamples*FSample)
faxis = deltaF*np.arange(len(St))
plt.plot(faxis, np.log10(np.abs(np.array([St, Sb]).T)))
See Wikipedia's explanation of spectral leakage and windowing for details on hh. The x-axis of the plot is calibrated in days. There are two thin peaks in the baseline spectrum at 1 and 2 day periodicity, and a broad peak around 1 day in the term1 spectrum.
It wasn't clear to me from your question if the baseline data already had the real data stripped out. If so, I think this shows that there isn't much structure in one that's distinguishable from the other.
Another thing to keep in mind is that Fourier analysis assumes the signal is stationary. Depending on the physics of what you're measuring this assumption may or may not be true. Certainly, the time-domain plot of term1 doesn't look at all stationary, with a wild change in character starting around day 12:
Having said all that, if you have a way to characterize the baseline data, you might be able to apply noise cancelation algorithms like the Widrow-Hoff LMS algorithm. Wikipedia presents a very theoretical overview, I'm not sure where to find a more practical application oriented explanation.
How did you come up with the term1/baseline separation in your example data?

It looks like both your "baseline signal" and your timeseries data contain information overlapping in spectral content. As such, an FFT doesn't provide any extra help in removing the overlapping portions that end up in the same FFT frequency bins.
As an FFT is a linear operator, subtracting in the frequency domain isn't any different from subtracting directly in the time domain.