I am having trouble with setting up a multivariate normal (MANOVA-like) regression in OpenBUGS on R version 3.3.2 (2016-10-31), Platform: x86_64-pc-linux-gnu (64-bit), Running under: Ubuntu 14.04.5 LTS.
My model is:
model
{
### Likelihood
for(i in 1:nfish){
CC[i, 1:ngen] ~ dmnorm(mu[i, 1:ngen], Sigma.inv[1:ngen, 1:ngen])
for(j in 1:ngen){
mu[i, j] <- CC.y[i] + beta0[j] + beta1[j] * Sdl[i]
}
}
### Priors
for(i in 1:ngen){
beta0[i] ~ dnorm(0.0, 0.01)
beta1[i] ~ dnorm(0.0, 0.01)
}
Sigma.inv[1:ngen, 1:ngen] ~ dwish(R[,], ngen)
Sigma[1:ngen, 1:ngen] <- inverse(Sigma.inv[,])
}
Data sample:
bugs.data <- list(CC = matrix(c(-3.75, -4.40, -1.60,
-3.75, -4.41, -1.71,
-8.67, NA, -5.23,
-8.92, -9.29, -5.12,
-8.69, -9.22, -9.48,
-9.87, -9.80, -18.32,
-4.71, -5.01, NA,
-4.69, -5.09, -1.67,
-3.62, -2.86, -0.26,
-3.93, -2.88, -0.21), ncol = 3),
CC.y = c(0.77, 0.82, 0.12, 0.08, 1.69, 1.69, -1.20, -1.15, 1.45, 1.46),
Sdl = c(0.96, 0.96, 0.76, 0.76, 0.40, 0.40, 0.97, 0.97, 0.27, 0.27),
nfish = 10,
ngen = 3,
R = matrix(c(3,0,0,0,3,0,0,0,3),ncol = 3));
Inits (not used since error occurs while compiling the model):
inits <- function() {list(beta0 = rnorm(3, -3.0, 1.0), beta1 = rnorm(3, 1.0, 0.2), Sigma.inv = matrix(rep(runif(1, 0, 1), 9), ncol = 3))}
Then when starting sampling:
> modelCheck("MVNmodel.bugs")
model is syntactically correct
> modelData("data.txt")
data loaded
> modelCompile(numChains = 3)
Error in handleRes(res) :
Internal "trap" error in OpenBUGS, or non-existent module or procedure called.
I get the same problem with interfacing openBUGS from R using BRugs or R2OpenBUGS. This is also not a problem of having NAs in the data. Note that the multivariate normal "Jaws" example from Lunn et al. (Jaws example) runs fine on my machine.
I am aware that this might be a hard problem to solve (See here), but I still hope for someone being able to help.
Thanks!
Related
We are currently facing a problem with a Prophet model which stops unexpectedly. We've used optuna as our HyperParameter model searching framework, and gave 400 trials per each TimeSeries (we are traininig Prophet on ~ 1000 Timeseries, 1 by 1, not all at once). For cross validation, we've used the methode from the Prophet class called cross_validation.
Here is the code we've used:
def get_best_hp_prophet_cv(df_for_cv, weeks_to_forecast):
logging.getLogger('prophet').setLevel(logging.ERROR)
logging.getLogger('fbprophet').setLevel(logging.ERROR)
# print('DF entered for search: ', df_for_cv)
cutoffs = generate_cutoffs(df_for_cv, weeks_to_forecast)
def objective(trial):
# print(cutoffs)
# print(df.tail(30))
param_grid = {
"changepoint_prior_scale": trial.suggest_categorical(
"changepoint_prior_scale", [0.001, 0.01, 0.1, 0.5, 0.05, 0.8, 0.9]
),
"seasonality_prior_scale": trial.suggest_categorical(
"seasonality_prior_scale", [0.01, 0.05, 0.1, 0.5, 1.0, 10]
),
"seasonality_mode": trial.suggest_categorical(
"seasonality_mode", ["multiplicative", "additive"]
),
"growth": trial.suggest_categorical("growth", ["linear"]),
"yearly_seasonality": trial.suggest_categorical(
"yearly_seasonality", [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16]
),
"daily_seasonality": trial.suggest_categorical("daily_seasonality",[False]),
"weekly_seasonality": trial.suggest_categorical("weekly_seasonality",[False]),
"uncertainty_samples": trial.suggest_categorical("uncertainty_samples",[0]),
}
prior_scale_month = trial.suggest_categorical('prior_scale_month', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
prior_scale_week_num = trial.suggest_categorical('prior_scale_week_num', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
prior_scale_avg_month_qty_over_df = trial.suggest_categorical('prior_scale_avg_month_qty_over_df', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
prior_scale_avg_week_nr_qty_over_df = trial.suggest_categorical('prior_scale_avg_week_nr_qty_over_df', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
# I ve used this only for testing to see if everything works fine
# param_grid = {
# 'changepoint_prior_scale': trial.suggest_categorical('changepoint_prior_scale', [0.001]),
# 'seasonality_prior_scale': trial.suggest_categorical('seasonality_prior_scale',[0.01, 0.1]),
# 'seasonality_mode' : trial.suggest_categorical('seasonality_mode',['additive']),
# 'growth': trial.suggest_categorical('growth',['linear']),
# 'yearly_seasonality': trial.suggest_categorical('yearly_seasonality',[14,15]),
# 'holidays_prior_scale' : trial.suggest_categorical('holidays_prior_scale',[10])
# }
# all_params = [dict(zip(param_grid.keys(), v)) for v in itertools.product(*param_grid.values())]
# mses = [] # Store the MSEs for each params here
# Use cross validation to evaluate all parameters
# for params in all_params:
m = Prophet(**param_grid)
m.add_regressor('month', prior_scale = prior_scale_month)
m.add_regressor('week_num', prior_scale = prior_scale_week_num)
m.add_regressor('avg_month_qty_over_df', prior_scale = prior_scale_avg_month_qty_over_df)
m.add_regressor('avg_week_nr_qty_over_df', prior_scale = prior_scale_avg_week_nr_qty_over_df)
m.fit(df_for_cv)
df_cv = cross_validation(
m, cutoffs=cutoffs, horizon="{} days".format(weeks_to_forecast*7), parallel="processes"
)
df_p = performance_metrics(df_cv, rolling_window=1)
return df_p["mse"].values[0]
# Find the best parameters
optuna_prophet = optuna.create_study(
direction="minimize", sampler=TPESampler(seed=321)
)
# * n_trials optuna hyperparameter.
#optuna_prophet.optimize(objective, n_trials=400)
optuna_prophet.optimize(objective, n_trials=1)
prophet_trial = optuna_prophet.best_trial
prophet_trial_params = prophet_trial.params
list_of_variables_outside_the_param_grid = ['prior_scale_month','prior_scale_week_num','prior_scale_avg_month_qty_over_df','prior_scale_avg_week_nr_qty_over_df']
params_outside_the_param_grid={}
param_grid = {}
for param_name in prophet_trial_params.keys():
if param_name in list_of_variables_outside_the_param_grid:
params_outside_the_param_grid.update({param_name : prophet_trial_params[param_name]})
else:
param_grid.update({param_name : prophet_trial_params[param_name]})
return param_grid, params_outside_the_param_grid
We've used Prophet before, but on less Timeseries (around 20) and with even more nr_of_trials in Optuna and never faced this issue before. Does anybody knows why is this happening? Has anybody a workaround? It always stops at the begging of the 54 trial from Optuna Search.
We've tried to run with only 1 trial on all 1000 Timeseries and it worked end - 2 - end, but still we can't figure it out why it stops on the 54th trial ?
Versions:
Optuna ---- 3.0.3.
Prophet ---- 1.1
I have an optimization problem of wagons repairs.
'Lon': 125, # London
'Ber': 175, # Berlin
'Maa': 225, # Maastricht
'Ams': 250, # Amsterdam
'Utr': 225, # Utrecht
'Hag': 200 # The Hague
}
Supply = {
'Arn': 600, # Arnhem
'Gou': 650 # Gouda
}
T = {
('Lon','Arn'): 1000,
('Lon','Gou'): 2.5,
('Ber','Arn'): 2.5,
('Ber','Gou'): 1000,
('Maa','Arn'): 1.6,
('Maa','Gou'): 2.0,
('Ams','Arn'): 1.4,
('Utr','Arn'): 0.8,
('Utr','Gou'): 1.0,
('Hag','Arn'): 1.4,
('Hag','Gou'): 0.8
}
T is a cost for all permitted routes. I defined the model and objective function.
# Step 0: Create an instance of the model
model = ConcreteModel()
model.dual = Suffix(direction=Suffix.IMPORT)
# Step 1: Define index sets
CUS = list(Demand.keys())
SRC = list(Supply.keys())
# Step 2: Define the decision
model.x = Var(list(T), domain = NonNegativeReals)
# Step 3: Define Objective
model.Cost = Objective(
expr = sum([T[i]*model.x[i] for i in model.x]),
sense = minimize)
But I have problems with constraints. If I am doing as shown below it throws an error.
# Step 4: Constraints
model.src = ConstraintList()
for s in SRC:
model.src.add(sum([model.x[c,s] for c in CUS]) <= Supply[s])
model.dmd = ConstraintList()
for c in CUS:
model.dmd.add(sum([model.x[c,s] for s in SRC]) == Demand[c])
Error is like that:
Maybe anyone knows how to fix the problem with constraints. To make it more flexible. I understand why it is an error because in T there are not all possible combinations, but it is right, some of the routes is restricted and I do not want to use them in optimization.
You can first get the possible ones like this:
# Step 4: Constraints
model.src = ConstraintList()
for s in SRC:
cposs = [t[0] for t in T.keys() if t[1] == s]
model.src.add(sum([model.x[c,s] for c in cposs]) <= Supply[s])
model.dmd = ConstraintList()
for c in CUS:
sposs = [t[1] for t in T.keys() if t[0] == c]
model.dmd.add(sum([model.x[c,s] for s in sposs]) == Demand[c])
I 'm using google colab to solve the homogeneous heat equation. I had made a program earlier with scipy using sparse matrices which worked upto N = 10(hyperparameter) but I need to run it for like N = 4... 1000 and thus it won't work on my pc. I therefore converted the code to tensorflow and here I 'm unable to use sparse matrices like I could in sympy but even the GPU/TPU computation is also slow and slower than my pc. Problems that I'm facing in the code and require solution for
1) tf.contrib is removed and thus I 've to use an older version of tensorflow for odeint function. Where is it in 2.0?
2)If the computation can be computed with sparse matrices it could be good since matrices are tridiagonal.I know about sparse_dense_mul() function but that returns dense tensor and it wouldn't do the job. The "func" function applies time independent boundary conditions and then requires matrix multiplication of (nxn) with (nX1) which gives (nX1) with multiple matrices.
Also the program was running faster without I created the class.
Also it's giving this
WARNING: Logging before flag parsing goes to stderr.
W0829 09:12:24.415445 139855355791232 lazy_loader.py:50]
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
* https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.
W0829 09:12:24.645356 139855355791232 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/integrate/python/ops/odes.py:233: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
when I run code for loop in range(2, 10) and tqdm does not display and cell keeps running forever but it works fine for in (2, 5) and tqdm bar does appears.
#find a way to use sparse matrices
class Heat:
def __init__(self, N):
self.N = N
self.H = 1/N
self.A = ts.to_dense(ts.SparseTensor(indices=[[0, 0], [0, 1]] + \
[[i, i+j] for i in range(1, N) for j in [-1, 0, 1]] +[[N, N-1], [N, N]],
values=self.H*np.array([1/3, 1/6] + [1/6, 2/3, 1/6]*(N-1) + [1/6, 1/3], dtype=np.float32),
dense_shape=(N+1, N+1 )))
self.D = ts.to_dense(ts.SparseTensor(indices=[[0, 0], [0, 1]] + [[i, i+j] \
for i in range(1, N) for j in [-1, 0, 1]] +[[N, N-1], [N, N]],
values=N*np.array([1-(1), -1 -(-1)] + [-1, 2, -1]*(N-1) + [-1-(-1), 1-(1)], dtype=np.float32),
dense_shape=(N+1, N+1)))
self.domain = tf.linspace(0.0, 1.0, N+1)
def f(k):
if k == 0:
return (1 + math.pi**2)*(math.pi*self.H - math.sin(math.pi*self.H))/(math.pi**2*self.H)
elif k == N:
return -(1 + math.pi**2)*(-math.pi*self.H + math.sin(math.pi*self.H))/(math.pi**2*self.H)
else:
return -2*(1 + math.pi**2)*(math.cos(math.pi*self.H) - 1)*math.sin(math.pi*self.H*k)/(math.pi**2*self.H)
self.F = tf.constant([f(k) for k in range(N+1)], shape=(N+1,), dtype=tf.float32) #caution! shape changed caution caution 1, N+1(problem) is different from N+1,
self.exact = tm.scalar_mul(scalar=np.exp(1), x=tf.sin(math.pi*self.domain))
def error(self):
return np.linalg.norm(self.exact.numpy() - self.approx, 2)
def func (self, y, t):
y = tf.Variable(y)
y = y[0].assign(0.0)
y = y[self.N].assign(0.0)
if self.N**2> 100:
y_dash = tl.matvec(tf.linalg.inv(self.A), tl.matvec(a=tm.negative(self.D), b=y, a_is_sparse=True) + tm.scalar_mul(scalar=math.exp(t), x=self.F)) #caution! shape changed F is (1, N+1) others too
else:
y_dash = tl.matvec(tf.linalg.inv(self.A), tl.matvec(a=tm.negative(self.D), b=y) + tm.scalar_mul(scalar=math.exp(t), x=self.F)) #caution! shape changed F is (1, N+1) others too
y_dash = tf.Variable(y_dash) #!!y_dash performs Hadamard product like multiplication not matrix-like multiplication;returns 2-D
y_dash = y_dash[0].assign(0.0)
y_dash = y_dash[self.N].assign(0.0)
return y_dash
def algo_1(self):
self.approx = tf.contrib.integrate.odeint(
func=self.func,
y0=tf.sin(tm.scalar_mul(scalar=math.pi, x=self.domain)),
t=tf.constant([0.0, 1.0]),
rtol=1e-06,
atol=1e-12,
method='dopri5',
options={"max_num_steps":10**10},
full_output=False,
name=None
).numpy()[1]
def algo_2(self):
self.approx = tf.contrib.integrate.odeint_fixed(
func=self.func,
y0=tf.sin(tm.scalar_mul(scalar=math.pi, x=self.domain)),
t=tf.constant([0.0, 1.0]),
dt=tf.constant([self.H**2], dtype=tf.float32),
method='rk4',
name=None
).numpy()[1]
df = pd.DataFrame(columns=["NumBasis", "Errors"])
Ns = [2**r for r in range(2, 10)]
l =[]
for i in tqdm_notebook(Ns):
heateqn = Heat(i)
heateqn.algo_1()
l.append([i, heateqn.error()])
df.append({"NumBasis":i, "Errors":heateqn.error()}, ignore_index=True)
tf.keras.backend.clear_session()
I'm running an mLSTM (multiplicative LSTM) transform (based on mLSTM by OpenAi (just the transform, it is already trained) but it takes a really long time to transform more than ~100,000 docs.
I want it to run on multiple GPUs. I saw some examples but I have no idea how to implement it on this mLSTM transform code.
The specific part that I want to run on multiple GPUs is:
def transform(xs):
tstart = time.time()
xs = [preprocess(x) for x in xs]
lens = np.asarray([len(x) for x in xs])
sorted_idxs = np.argsort(lens)
unsort_idxs = np.argsort(sorted_idxs)
sorted_xs = [xs[i] for i in sorted_idxs]
maxlen = np.max(lens)
offset = 0
n = len(xs)
smb = np.zeros((2, n, hps.nhidden), dtype=np.float32)
for step in range(0, ceil_round_step(maxlen, nsteps), nsteps):
start = step
end = step+nsteps
xsubseq = [x[start:end] for x in sorted_xs]
ndone = sum([x == b'' for x in xsubseq])
offset += ndone
xsubseq = xsubseq[ndone:]
sorted_xs = sorted_xs[ndone:]
nsubseq = len(xsubseq)
xmb, mmb = batch_pad(xsubseq, nsubseq, nsteps)
for batch in range(0, nsubseq, nbatch):
start = batch
end = batch+nbatch
batch_smb = seq_rep(
xmb[start:end], mmb[start:end],
smb[:, offset+start:offset+end, :])
smb[:, offset+start:offset+end, :] = batch_smb
features = smb[0, unsort_idxs, :]
print('%0.3f seconds to transform %d examples' %
(time.time() - tstart, n))
return features
This is just a snippet of the full code (I don't think it's OK to copy the entire code here).
The part you're referring to is not the place that splits the computation across GPUs, it only transforms the data (on a CPU!) and runs the session.
The correct place is one that defines the computational graph, e.g. mlstm method. There are many ways to split graph, e.g. place LSTM cells on different GPUs, so that the input sequence can be processed in parallel:
def mlstm(inputs, c, h, M, ndim, scope='lstm', wn=False):
[...]
for idx, x in enumerate(inputs):
with tf.device('/gpu:' + str(i % GPU_COUNT)):
m = tf.matmul(x, wmx) * tf.matmul(h, wmh)
z = tf.matmul(x, wx) + tf.matmul(m, wh) + b
[...]
By the way, there is a useful config option in tensorflow log_device_placement that helps to see the execution details in the output. Here's an example:
import tensorflow as tf
# Creates a graph.
with tf.device('/gpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='b')
c = tf.add(a, b)
# Creates a session with log_device_placement set to True.
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:
# Prints the following:
# Device mapping:
# /job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: <GPU name>, pci bus id: 0000:01:00.0, compute capability: 6.1
# Add: (Add): /job:localhost/replica:0/task:0/device:GPU:0
# b: (Const): /job:localhost/replica:0/task:0/device:GPU:0
# a: (Const): /job:localhost/replica:0/task:0/device:GPU:0
print(sess.run(c))
Currently going through examples volume 1 and came across an error with the dyes example.
When I try to load inits from the example it returns "this chain contains uninitialized variables. I am not sure which part of it is not right as on the first sight I see theta, tau.btw and tau.with is all specified and nothing is left out.
I am using the code directly from Examples Vol 1 under help tab. The same error happened to all three choices of priors for between-variation.
I would really appreciate any advice on the problem. Thanks in advance.
Below is the code I copied directly from the dyes example.
model
{
for( i in 1 : batches ) {
mu[i] ~ dnorm(theta, tau.btw)
for( j in 1 : samples ) {
y[i , j] ~ dnorm(mu[i], tau.with)
}
}
theta ~ dnorm(0.0, 1.0E-10)
# prior for within-variation
sigma2.with <- 1 / tau.with
tau.with ~ dgamma(0.001, 0.001)
# Choice of priors for between-variation
# Prior 1: uniform on SD
#sigma.btw~ dunif(0,100)
#sigma2.btw<-sigma.btw*sigma.btw
#tau.btw<-1/sigma2.btw
# Prior 2: Uniform on intra-class correlation coefficient,
# ICC=sigma2.btw / (sigma2.btw+sigma2.with)
ICC ~ dunif(0,1)
sigma2.btw <- sigma2.with *ICC/(1-ICC)
tau.btw<-1/sigma2.btw
# Prior 3: gamma(0.001, 0.001) NOT RECOMMENDED
#tau.btw ~ dgamma(0.001, 0.001)
#sigma2.btw <- 1 / tau.btw
}
Data
list(batches = 6, samples = 5,
y = structure(
.Data = c(1545, 1440, 1440, 1520, 1580,
1540, 1555, 1490, 1560, 1495,
1595, 1550, 1605, 1510, 1560,
1445, 1440, 1595, 1465, 1545,
1595, 1630, 1515, 1635, 1625,
1520, 1455, 1450, 1480, 1445), .Dim = c(6, 5)))
Inits1
list(theta=1500, tau.with=1, sigma.btw=1)
Inits2
list(theta=1500, tau.with=1,ICC=0.5)
Inits3
list(theta=1500, tau.with=1, tau.btw=1)
That is not an error per se. Yes you have provided the inits for the parameters of interest.
However there are the six mu[i] variables that are not data, but are variables drawn from mu[i] ~ dnorm(theta, tau.btw).
You could provide initial values for these as well, but it is best imo to just click on gen inits if you are using WinBUGS from the GUI - this will provide initial values for those.