It there a way to cache OpenMDAO component outputs to avoid duplicate executions? - optimization

I am writing a model consists of five subsystems. The first subsystem is generates data for other subsystems using the inputs it does not solve anything iteratively, therefore its outputs not changed during computation. I want it calls once the compute method just like the initialization. How can I write a model that calls once in run_model and calls every time just once in run_driver?

It's a little hard to be sure without more details, but you mention "iterative" so Im guessing you have a solver at the top level of your model and there is a component not involved in that solver loop, but that is getting called each time the solver iterates.
The solution to this is to make a sub-group in your model that has just the components that need to iterate. Put your only-run-once component at the top of the model, along with that group. Put the iterative solver on the sub-group.
An alternative solution is to add a bit of caching to your component, so it checks its inputs to see if they have changed. If they have, re-run. If they have not, just keep the old answer.
Here is an example that includes both features (note: the solver in this example does not converge because its a toy problem that doesn't have a valid physical solution. I just threw it together to illustrate the model structure and caching)
import openmdao.api as om
# from openmdao.utils.assert_utils import assert_check_totals
class StingyComp(om.ExplicitComponent):
def setup(self):
self.add_input('x1', val=2.)
self.add_input('x2', val=3.)
self.add_output('x')
self._input_hash = None
def compute(self, inputs, outputs):
x1 = inputs['x1'][0] # pull the scalar out so you can hash it
x2 = inputs['x2'][0]
print("running StingyComp")
current_input_hash = hash((x1, x2))
if self._input_hash != current_input_hash :
print(' ran compute')
outputs['x'] = 2*x1 + x2**2
self._input_hash = current_input_hash
else:
print(' skipped compute')
class NormalComp(om.ExplicitComponent):
def setup(self):
self.add_input('x1', val=2.)
self.add_input('x2', val=3.)
self.add_output('y')
def compute(self, inputs, outputs):
x1 = inputs['x1']
x2 = inputs['x2']
print("running normal Comp")
outputs['y'] = x1 + x2
p = om.Problem()
p.model.add_subsystem('run_once1', NormalComp(), promotes=['*'])
p.model.add_subsystem('run_once2', StingyComp(), promotes=['*'])
sub_group = p.model.add_subsystem('sub_group', om.Group(), promotes=['*']) # transparent group that could hold sub-solver
sub_group.add_subsystem('C1', om.ExecComp('f1 = f2**2 + 1.5 * x - y**2.5'), promotes=['*'])
sub_group.add_subsystem('C2', om.ExecComp('f2 = f1**2 + x**1.5 - 2.5*y'), promotes=['*'])
sub_group.nonlinear_solver = om.NewtonSolver(solve_subsystems=False)
sub_group.linear_solver = om.DirectSolver()
p.setup()
print('first run')
p.run_model()
print('second run, same inputs')
p.run_model()
p['x1'] = 10
p['x2'] = 27.5
print('third run, new inputs')
p.run_model()

Related

In OpenMDAO's ExecComp, is shape_by_conn compatible with has_diag_partials?

I have an om.ExecComp that performs a simple operation:
"d_sq = x**2 + y**2"
where x, y, and d_sq are always 1D np.arrays. I'd like to be able to use this with large arrays without allocating a large dense matrix. I'd also like the length of the array to be configured based on the shape of the connections.
However, if I specify x={"shape_by_conn": True} rather than x={"shape":100000}, even if I also have has_diag_partials=True, it attempts to allocate a 100000^2 array. Is there a way to make these two options compatible?
First, I'll note that you're using ExecComp a bit outside its design intended purpose. Thats not to say that you're something totally invalid, but generally speaking ExecComp was designed for small, cheap calculations. Passing it giant arrays is not something we test for.
That being said, I think what you want will work. When you use shape_by_conn in this you need to be sure to size both your inputs and outputs. I've provided an example, along with a manually defined component that does the same thing. Since your equations are pretty simple, this would be a little faster overall.
import numpy as np
import openmdao.api as om
class SparseCalc(om.ExplicitComponent):
def setup(self):
self.add_input('x', shape_by_conn=True)
self.add_input('y', shape_by_conn=True)
self.add_output('d_sq', shape_by_conn=True, copy_shape='x')
def setup_partials(self):
# when using shape_by_conn, you need to delcare partials
# in this secondary method
md = self.get_io_metadata(iotypes='input')
# everything should be the same shape, so just need this one
x_shape = md['x']['shape']
row_col = np.arange(x_shape[0])
self.declare_partials('d_sq', 'x', rows=row_col, cols=row_col)
self.declare_partials('d_sq', 'y', rows=row_col, cols=row_col)
def compute(self, inputs, outputs):
outputs['d_sq'] = inputs['x']**2 + inputs['y']**2
def compute_partials(self, inputs, J):
J['d_sq', 'x'] = 2*inputs['x']
J['d_sq', 'y'] = 2*inputs['y']
if __name__ == "__main__":
p = om.Problem()
# use IVC here, because you have to have something connected to
# in order to use shape_by_conn. Normally IVC is not needed
ivc = p.model.add_subsystem('ivc', om.IndepVarComp(), promotes=['*'])
ivc.add_output('x', 3*np.ones(10))
ivc.add_output('y', 2*np.ones(10))
# p.model.add_subsystem('sparse_calc', SparseCalc(), promotes=['*'])
p.model.add_subsystem('sparse_exec_calc',
om.ExecComp('d_sq = x**2 + y**2',
x={'shape_by_conn':True},
y={'shape_by_conn':True},
d_sq={'shape_by_conn':True,
'copy_shape':'x'},
has_diag_partials=True),
promotes=['*'])
p.setup(force_alloc_complex=True)
p.run_model()
If you still find this isn't working as expected, please feel free to submit a bug report with a test case that shows the problem clearly (i.e. will raise the error you're seeing). In this case, the provided manual component can serve as a workaround.

Why Assignment Operation cannot be added as a parameter to the control depedency in tensorflow?

I have a simple model. But I need to modify a variable before performing an update. Hence, I have the following:
l = tf.Variable(tf.constant(0.01), trainable=False, name="l")
baseline = tf.Variable(tf.constant(0.0), dtype=tf.float32, name="baseline")
# Actor optimizer
# Note that our target is: e^{-L} where L is the loss on the validation dataset.
# So we pass to the target mae_loss in the code below
def optimize_actor(scores_a, scores_v, target):
with tf.name_scope('Actor_Optimizer'):
update_moving_loss = tf.assign(baseline, l * baseline + (1 - l) * target, name="update_baseline")
dec_l = l.assign_add(0.001)
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies([update_ops, update_moving_loss, dec_l]):
loss_a = scores_a * tf.exp(target - baseline)
loss_v = scores_v * tf.exp(target - baseline)
actor_a_optimizer = tf.train.AdamOptimizer(0.0001).minimize(loss_a)
actor_v_optimizer = tf.train.AdamOptimizer(0.0001).minimize(loss_v)
return actor_a_optimizer, actor_v_optimizer
Therefore, when running the above script, I got the following error:
Can not convert a list into a Tensor or Operation.
What is causing this problem is update_moving_loss and dec_l. When I remove them, the code runs fine.
Please note that I am using tensorflow 1.7
Any help is much appreciated!!!
I cannot reproduce your problem, but my guess is the problem lies with using tf.get_collection. As specified in the documentation, it will return a list of values in the collection. Let's call this list L to make things concise.
You then do something like this:
L = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies([L, tensor1, tensor2]):
...
The argument there is another list, with three elements: your list L from before, and two tensors. This is the problem: the first element is a list, and thus you see
Can not convert a list into a Tensor or Operation.
You can probably resolve the issue by appending to L rather than making another list-with-a-list-inside:
L = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies( L + [tensor1, tensor2]):
...
The + operator adds the new list elements to the original list L.

How to simulate from priors with pymc3

I'd like to simulate y from the prior (not from the posterior) with pymc3.
I first defined the model:
import pymc3 as pm
with pm.Model() as m:
mu = pm.Normal('mu', mu=0, sd=10)
sigma = pm.Uniform('sigma', lower=0, upper=10)
y = pm.Normal('y', mu=mu, sd=sigma)
trace = pm.sample(1000, tune=1000)
Then I tried to get 10 simulated y from the model with:
y_pred = pm.sample_ppc(trace, 10, m, size=10)
But result comes out empty. I searched through the documentation but I didn't find a relevant example. Is it possible to do it with pymc3?
The trace contains the sample from the prior when no observed is associated with the model definition. However, this could fail sometimes. We are currently working on a sample_prior function that would make this process easier and more straightforward: https://github.com/pymc-devs/pymc3/pull/2876

TensorFlow: How to apply the same image distortion to multiple images

Starting from the Tensorflow CNN example, I'm trying to modify the model to have multiple images as an input (so that the input has not just 3 input channels, but multiples of 3 by stacking images).
To augment the input, I try to use random image operations, such as flipping, contrast and brightness provided in TensorFlow.
My current solution to apply the same random distortion to all input images is to use a fixed seed value for these operations:
def distort_image(image):
flipped_image = tf.image.random_flip_left_right(image, seed=42)
contrast_image = tf.image.random_contrast(flipped_image, lower=0.2, upper=1.8, seed=43)
brightness_image = tf.image.random_brightness(contrast_image, max_delta=0.2, seed=44)
return brightness_image
This method is called multiple times for each image at graph construction time, so I thought for each image it will use the same random number sequence and consequently, it will result in have the same applied image operations for my image input sequence.
# ...
# distort images
distorted_prediction = distort_image(seq_record.prediction)
distorted_input = []
for i in xrange(INPUT_SEQ_LENGTH):
distorted_input.append(distort_image(seq_record.input[i,:,:,:]))
stacked_distorted_input = tf.concat(2, distorted_input)
# Ensure that the random shuffling has good mixing properties.
min_queue_examples = int(num_examples_per_epoch *
MIN_FRACTION_EXAMPLES_IN_QUEUE)
# Generate a batch of sequences and prediction by building up a queue of examples.
return generate_sequence_batch(stacked_distorted_input, distorted_prediction, min_queue_examples,
batch_size, shuffle=True)
In theory, this works fine. And after doing some test runs, this really seemed to solve my problem. But after a while, I found out that I'm having a race-condition, because I use the input pipeline of the CNN-example code with multiple threads (which is the suggested method in TensorFlow to improve performance and reduce memory consumption at runtime):
def generate_sequence_batch(sequence_in, prediction, min_queue_examples,
batch_size):
num_preprocess_threads = 8 # <-- !!!
sequence_batch, prediction_batch = tf.train.shuffle_batch(
[sequence_in, prediction],
batch_size=batch_size,
num_threads=num_preprocess_threads,
capacity=min_queue_examples + 3 * batch_size,
min_after_dequeue=min_queue_examples)
return sequence_batch, prediction_batch
Because multiple threads create my examples, it is not guaranteed anymore that all image operations are performed in the right order (in sense of the right order of random operations).
Here I came to a point where I got completely stuck. Does anyone know how to solve this problem to apply the same image distortion to multiple images?
Some thoughts of mine:
I thought about to do some synchronizations arround these image distortion methods, but I could find anything provided by TensorFlow
I tried to generate to generate a random number for e.g. the random brightness delta using tf.random_uniform() by myself and use this value for tf.image.adjust_contrast(). But the result of the TensorFlow random generator is always a tensor, and I have not found a way to use this tensor as a parameter for tf.image.adjust_contrast() which expects a simple float32 for its contrast_factor parameter.
A solution that would (partly) work would be to combine all images to a huge image using tf.concat(), apply random operations to change contrast and brightness, and split the image afterwards. But this would not work for random flipping, because this would (at least in my case) change the order of the images, and there is no way to detect whether tf.image.random_flip_left_right() has performed a flip or not, which would be required to fix the wrong order of images if necessary.
Here is what I came up with by looking at the code of random_flip_up_down and random_flip_left_right within tensorflow :
def image_distortions(image, distortions):
distort_left_right_random = distortions[0]
mirror = tf.less(tf.pack([1.0, distort_left_right_random, 1.0]), 0.5)
image = tf.reverse(image, mirror)
distort_up_down_random = distortions[1]
mirror = tf.less(tf.pack([distort_up_down_random, 1.0, 1.0]), 0.5)
image = tf.reverse(image, mirror)
return image
distortions = tf.random_uniform([2], 0, 1.0, dtype=tf.float32)
image = image_distortions(image, distortions)
label = image_distortions(label, distortions)
I would do something like this using tf.case. It allows you to specify what to return if certain condition holds https://www.tensorflow.org/api_docs/python/tf/case
import tensorflow as tf
def distort(image, x):
# flip vertically, horizontally, both, or do nothing
image = tf.case({
tf.equal(x,0): lambda: tf.reverse(image,[0]),
tf.equal(x,1): lambda: tf.reverse(image,[1]),
tf.equal(x,2): lambda: tf.reverse(image,[0,1]),
}, default=lambda: image, exclusive=True)
return image
def random_distortion(image):
x = tf.random_uniform([1], 0, 4, dtype=tf.int32)
return distort(image, x[0])
To check if it works.
import numpy as np
import matplotlib.pyplot as plt
# create image
image = np.zeros((25,25))
image[:10,5:10] = 1.
# create subplots
fig, axes = plt.subplots(2,2)
for i in axes.flatten(): i.axis('off')
with tf.Session() as sess:
for i in range(4):
distorted_img = sess.run(distort(image, i))
axes[i % 2][i // 2].imshow(distorted_img, cmap='gray')
plt.show()

Derivative check with scalers

I have a problem that I want to scale the design variables. I have added the scaler, but I want to check the derivative to make sure it is doing what I want it to do. Is there a way to check the scaled derivative? I have tried to use check_total_derivatives() but the derivative is the exact same regardless of what value I put for scaler:
from openmdao.api import Component, Group, Problem, IndepVarComp, ExecComp
from openmdao.drivers.pyoptsparse_driver import pyOptSparseDriver
class Scaling(Component):
def __init__(self):
super(Scaling, self).__init__()
self.add_param('x', shape=1)
self.add_output('y', shape=1)
def solve_nonlinear(self, params, unknowns, resids):
unknowns['y'] = 1000. * params['x']**2 + 2
def linearize(self, params, unknowns, resids):
J = {}
J['y', 'x'] = 2000. * params['x']
return J
class ScalingGroup(Group):
def __init__(self,):
super(ScalingGroup, self).__init__()
self.add('x', IndepVarComp('x', 0.0), promotes=['*'])
self.add('g', Scaling(), promotes=['*'])
p = Problem()
p.root = ScalingGroup()
# p.driver = pyOptSparseDriver()
# p.driver.options['optimizer'] = 'SNOPT'
p.driver.add_desvar('x', lower=0.005, upper=100., scaler=1000)
p.driver.add_objective('y')
p.setup()
p['x'] = 3.
p.run()
total = p.check_total_derivatives()
# Derivative is the same regardless of what the scaler is.
The scalers and adders are consistent in their behavior, so the check derivatives routines give results in unscaled terms to be more intuitive.
If you really want to see what impact the scaler is having when the NLP sees the scaled value and you're using SNOPT, you can add SNOPT's derivative check capability:
p.driver.opt_settings['Verify level'] = 3
SNOPT_print.out will contain, with the scaler set to 1:
Column x(j) dx(j) Element no. Row Derivative Difference approxn
1 3.00000000E+00 2.19E-06 Objective 6.00000000E+03 6.00000219E+03 ok
Or if we change it to the x scaler to 1000:
Column x(j) dx(j) Element no. Row Derivative Difference approxn
1 3.00000000E+03 1.64E-03 Objective 6.00000000E+00 6.00000164E+00 ok
So in the units of the problem, which check_total_derivatives uses, the derivative doesn't change. But the scaled value as seen by the optimizer is changing.
Another way to see exactly what the optimizer is seeing from calc_gradient is to mimic the call to calc_gradient. This is not necessarily easy to figure out, but I thought I would paste it here for reference.
print p.calc_gradient(list(p.driver.get_desvars().keys()),
list(p.driver.get_objectives().keys()) + list(p.driver.get_constraints().keys()),
dv_scale=p.driver.dv_conversions,
cn_scale=p.driver.fn_conversions)