How to map different indices in Pyomo? - optimization

I am a new Pyomo/Python user. Now I need to formulate one set of constraints with index 'n', where all of the 3 components are with different indices but correlate with index 'n'. I am just curious that how I can map the relationship between these sets.
In my case, I read csv files in which their indices are related to 'n' to generate my set. For example: a1.n1, a2.n3, a3.n5 /// b1.n2, b2.n4, b3.n6, b4.n7 /// c1.n1, c2.n2, c3.n4, c4.n6 ///. The constraint expression of index n1 and n2 is the follows for example:
for n1: P(a1.n1) + L(c1.n1) == D(n1)
for n2: - F(b1.n2) + L(c2.n2) == D(n2)
Now let's go the coding. The set creating codes are as follow, they are within a class:
import pyomo
import pandas
import pyomo.opt
import pyomo.environ as pe
class MyModel:
def __init__(self, Afile, Bfile, Cfile):
self.A_data = pandas.read_csv(Afile)
self.A_data.set_index(['a'], inplace = True)
self.A_data.sort_index(inplace = True)
self.A_set = self.A_data.index.unique()
... ...
Then I tried to map the relationship in the constraint construction like follows:
def createModel(self):
self.m = pe.ConcreteModel()
self.m.A_set = pe.Set( initialize = self.A_set )
def obj_rule(m):
return ...
self.m.OBJ = pe.Objective(rule = obj_rule, sense = pe.minimize)
def constr(m, n)
As = self.A_data.reset_index()
Amap = As[ As['n'] == n ]['a']
Bs = self.B_data.reset_index()
Bmap = Bs[ Bs['n'] == n ]['b']
Cs = self.C_data.reset_index()
Cmap = Cs[ Cs['n'] == n ]['c']
return sum(m.P[(p,n)] for p in Amap) - sum(m.F[(s,n)] for s in Bmap) + sum(m.L[(r,n)] for r in Cmap) == self.D_data.ix[n, 'D']
self.m.cons = pe.Constraint(self.m.D_set, rule = constr)
def solve(self):
... ...
Finally, the error raises when I run this:
KeyError: "Index '(1, 1)' is not valid for indexed component 'P'"
I know it is the wrong way, so I am wondering if there is a good way to map their relationships. Thanks in advance!
Gabriel

I just forgot to post my answer to my own question when I solved this one week ago. The key thing towards this problem is setting up a map index.
Let me just modify the code in the question. Firstly, we need to modify the dataframe to include the information of the mapped indices. Then, the set for the mapped index can be constructed, taking 2 mapped indices as example:
self.m.A_set = pe.Set( initialize = self.A_set, dimen = 2 )
The names of the two mapped indices are 'alpha' and 'beta' respectively. Then the constraint can be formulated, based on the variables declared at the beginning:
def constr(m, n)
Amap = self.A_data[ self.A_data['alpha'] == n ]['beta']
Bmap = self.B_data[ self.B_data['alpha'] == n ]['beta']
return sum(m.P[(i,n)] for i in Amap) + sum(m.L[(r,n)] for r in Bmap) == D.loc[n, 'D']
m.TravelingBal = pe.Constraint(m.A_set, rule = constr)
The summation groups all associated B to A with a mapped index set.

Related

Is there a method for converting a winmids object to a mids object?

Suppose I create 10 multiply-imputed datasets and use the (wonderful) MatchThem package in R to create weights for my exposure variable. The MatchThem package takes a mids object and converts it to an object of the class winmids.
My desired output is a mids object - but with weights. I hope to pass this mids object to BRMS as follows:
library(brms)
m0 <- brm_multiple(Y|weights(weights) ~ A, data = mids_data)
Open to suggestions.
EDIT: Noah's solution below will unfortunately not work.
The package's first author, Farhad Pishgar, sent me the following elegant solution. It will create a mids object from a winmidsobject. Thank you Farhad!
library(mice)
library(MatchThem)
#"weighted.dataset" is our .wimids object
#Extracting the original dataset with missing value
maindataset <- complete(weighted.datasets, action = 0)
#Some spit-and-polish
maindataset <- data.frame(.imp = 0, .id = seq_len(nrow(maindataset)), maindataset)
#Extracting imputed-weighted datasets in the long format
alldataset <- complete(weighted.datasets, action = "long")
#Binding them together
alldataset <- rbind(maindataset, alldataset)
#Converting to .mids
newmids <- as.mids(alldataset)
Additionally, for BRMS, I worked out this solution which instead creates a list of dataframes. It will work in fewer steps.
library("mice")
library("dplyr")
library("MatchThem")
library("brms") # for bayesian estimation.
# Note, I realise that my approach here is not fully Bayesian, but that is a good thing! I need to ensure balance in the exposure.
# impute missing data
data("nhanes2")
imp <- mice(nhanes2, printFlag = FALSE, seed = 0, m = 10)
# MathThem. This is just a fast method
w_imp <- weightthem(hyp ~ chl + age, data = imp,
approach = "within",
estimand = "ATE",
method = "ps")
# get individual data frames with weights
out <- complete(w_imp, action ="long", include = FALSE, mild = TRUE)
# assemble individual data frames into a list
m <- 10
listdat<- list()
for (i in 1:m) {
listdat[[i]] <- as.data.frame(out[[i]])
}
# pass the list to brms, and it runs as it should!
fit_1 <- brm_multiple(bmi|weights(weights) ~ age + hyp + chl,
data = listdat,
backend = "cmdstanr",
family = "gaussian",
set_prior('normal(0, 1)',
class = 'b'))
brm_multiple() can take in a list of data frames for its data argument. You can produce this from the wimids object using complete(). The output of complete() with action = "all" is a mild object, which is a list of data frames, but this is not recognized by brm_multiple() as such. So, you can just convert it to a list. This should look like the following:
df_list <- complete(mids_data, "all")
class(df_list) <- "list"
m0 <- brm_multiple(Y|weights(weights) ~ A, data = df_list)
Using complete() automatically adds a weights column to the resulting imputed data frames.

How to change the mutable parameter in Pyomo (AbstractModel)?

I am trying to update my mutable parameter Nc in my Abstract model
the initial value is 4
I constructed the instance then change instance.Nc to 5 and solve it but it is still 4 (initial value) , can any body help ?
from pyomo.environ import *
import random
model = AbstractModel()
model.i = RangeSet(40)
model.j = Set(initialize=model.i)
model.x = Var(model.i,model.j, initialize=0,within=Binary)
model.y = Var(model.i, within=Binary)
model.Nc=Param(initialize=5,mutable=True)
def Ninit(model,i):
return random.randint(0,1)
model.N=Param(model.i,initialize=Ninit,mutable=True)
def Dinit(model,i,j):
return random.random()
model.D=Param(model.i,model.j,initialize=Dinit,mutable=True)
def rule_C1(model,i,j):
return model.x[i,j]<=model.N[i]*model.y[j]
model.C1 = Constraint(model.i,model.j,rule=rule_C1)
def rule_C2(model):
return sum(model.y[i] for i in model.i )==model.Nc
model.C2 = Constraint(rule=rule_C2)
def rule_C3(model,i):
return sum(model.x[i,j] for j in model.j)==model.N[i]
model.C3 = Constraint(model.i,rule=rule_C3)
def rule_OF(model):
return sum( model.x[i,j]*model.D[i,j] for i in model.i for j in model.j )
model.obj = Objective(rule=rule_OF, sense=minimize)
opt = SolverFactory('glpk')
#model.NC=4
instance = model.create_instance()
instance.NC=4
results = opt.solve(instance) # solves and updates instance
print('NC= ',value(instance.Nc))
print('OF= ',value(instance.obj))
It seems you are actually initializing your parmeter Nc to 5 (model.Nc=Param(initialize=5,mutable=True)) and then changing it to 4 once you create the instance (instance.Nc=4), so you might want to do the opposite (model.Nc=Param(initialize=4,mutable=True) then instance.Nc=4)
Also, note that you are inconsistantly addressing the Nc parameter throughout the code. When you declare the parameter you name it "Nc" (model.Nc=Param(initialize=5,mutable=True)), which is the actual python variable that Pyomo will use in the model, but later you try to change it with capital letters "NC", which is not a parameter (instance.NC=4). Minor typos like these can cause confusion and give you errors. Make sure to fix them and give it a try again

Pyomo: How to write a constraint for each (i,j) pair

Edit: I have also added a constraint 1.5 to illustrate maybe a different way of approaching the constraint.
I am trying to write the following constraints in Pyomo for each (i,j) pair on an MxN grid:
The code that I have thus far is as follows, and I am just hoping I can get some feedback on whether or not the constraint definition is written properly to meet the intention.The idea is that each (i,j) cell on the 6x6 grid will have the following two constraints.
model = AbstractModel()
#Define the index sets for the grid, time horizions, and age classes:
model.Iset = RangeSet(6)
model.Jset = RangeSet(6)
model.Tset = RangeSet(7)
model.Kset = RangeSet(50)
#Define model parameters:
model.s = Param(within=NonNegativeIntegers)
#Define model variables:
model.juvenille = Var(model.Iset, model.Jset, model.Tset, model.Kset,
within=NonNegativeReals, initialize = "some expression"
#Constraints:
# Constraint #1
def juv_advance(model, i, j, t, k):
return model.juvenille[i,j,t+1,k+1] == model.juvenille[i,j,t,k]*model.juvsurv
# Constraint #1.5
def juv_advance(model, t, k):
return model.juvenille[t+1,k+1] == model.juvenille[t,k]*model.s \\
for i in model.Iset for j in model.Jset
# Constraint #2
def juv_total(model, i, j, t, k):
return sum(model.juvenille[k] for k in range(1,50))
Additionally, if anybody feels like answering this... how could you save the calculated j_t+1 value to use as the initial value in the next time period.
I would try something like this:
model = AbstractModel()
#Define the index sets for the grid, time horizions, and age classes:
model.Iset = RangeSet(6)
model.Jset = RangeSet(6)
model.Tset = RangeSet(7)
model.Kset = RangeSet(50)
#Define model parameters:
model.s = Param(within=NonNegativeIntegers)
#Define model variables:
model.juvenille = Var(model.Iset, model.Jset, model.Tset, model.Kset,
within=NonNegativeReals, initialize="some expression")
# As far as I see your problem in you second constraint the big J is a new variable ?
If that is the case than you have to create it:
model.J_big =Var(model.Iset, model.Jset, model.Tset, within=NonNegativeReals)
#Constraints:
# Constraint #1
def juv_advance(model, i, j, t, k):
k_len = len(model.Kset)
t_len = len(model.Tset)
if k == 1 and t == 1:
return "some expression"
elif t < t_len and k < k_len:
return model.juvenille[i,j,t+1,k+1] == model.juvenille[i,j,t,k]*model.s
else:
return "Here has to come a statement what should happen with the last index (because if you are iterating to k=50 there is no k=51) "
model.ConstraintNumber1 = Constraint(model.Iset, model.Jset, model.Tset, model.Kset, rule=juv_advance)
# Constraint #2
def juv_total(model, i, j, t, k):
return model.J_big[i,k,j] == sum(model.juvenille[i,j,t,k] for k in model.Kset)
model.ConstraintNumber2 = Constraint(model.Iset, model.Jset, model.Tset, rule=juv_total)
It is important that you not only define the rule of the constraint but also the constraint itself. Also you have to have in mind that you K and T sets are ending somewhere and that an expression of k+1 does not work if there is no k+1. Another point that could be mentioned is that if you start with k+1 == something the first k-value that is taken into account is k = 2.
I hope this helps, maybe someone also knows something smarter, I am also quite new to pyomo.

Retrieve indices for rows of a PyTables table matching a condition using `Table.where()`

I need the indices (as numpy array) of the rows matching a given condition in a table (with billions of rows) and this is the line I currently use in my code, which works, but is quite ugly:
indices = np.array([row.nrow for row in the_table.where("foo == 42")])
It also takes half a minute, and I'm sure that the list creation is one of the reasons why.
I could not find an elegant solution yet and I'm still struggling with the pytables docs, so does anybody know any magical way to do this more beautifully and maybe also a bit faster? Maybe there is special query keyword I am missing, since I have the feeling that pytables should be able to return the matched rows indices as numpy array.
tables.Table.get_where_list() gives indices of the rows matching a given condition
I read the source of pytables, where() is implemented in Cython, but it seems not fast enough. Here is a complex method that can speedup:
Create some data first:
from tables import *
import numpy as np
class Particle(IsDescription):
name = StringCol(16) # 16-character String
idnumber = Int64Col() # Signed 64-bit integer
ADCcount = UInt16Col() # Unsigned short integer
TDCcount = UInt8Col() # unsigned byte
grid_i = Int32Col() # 32-bit integer
grid_j = Int32Col() # 32-bit integer
pressure = Float32Col() # float (single-precision)
energy = Float64Col() # double (double-precision)
h5file = open_file("tutorial1.h5", mode = "w", title = "Test file")
group = h5file.create_group("/", 'detector', 'Detector information')
table = h5file.create_table(group, 'readout', Particle, "Readout example")
particle = table.row
for i in range(1001000):
particle['name'] = 'Particle: %6d' % (i)
particle['TDCcount'] = i % 256
particle['ADCcount'] = (i * 256) % (1 << 16)
particle['grid_i'] = i
particle['grid_j'] = 10 - i
particle['pressure'] = float(i*i)
particle['energy'] = float(particle['pressure'] ** 4)
particle['idnumber'] = i * (2 ** 34)
# Insert a new particle record
particle.append()
table.flush()
h5file.close()
Read the column in chunks and append the indices into a list and concatenate the list to array finally. You can change the chunk size according to your memory size:
h5file = open_file("tutorial1.h5")
table = h5file.get_node("/detector/readout")
size = 10000
col = "energy"
buf = np.zeros(batch, dtype=table.coldtypes[col])
res = []
for start in range(0, table.nrows, size):
length = min(size, table.nrows - start)
data = table.read(start, start + batch, field=col, out=buf[:length])
tmp = np.where(data > 10000)[0]
tmp += start
res.append(tmp)
res = np.concatenate(res)

QGIS - Retrieve start and end features from a line

I'm creating lines by selecting two features from various layers. When I create a line a form pops up. In this form I want to display data from the start and end features of the line.
What I'm currently doing is retrieving the vertices as point :
geom = feature.geometry ()
line = geom.asPolyline ()
pointFather = ligne[0]
pointChild = ligne[-1]
then I get the coordinates of each point :
xf = pointFather.x()
yf = pointFather.y()
and then I look into each possible layer to find the features with the same coordinates, just to retrieve the features I just clicked on !
for layer in layerList:
provider = layer.dataProvider()
iter = provider.getFeatures()
for feature in iter:
geom = feature.geometry().asPoint()
if geom.x() == xf and geom.y() == yf:
It must be something easier to do to directly retrieve the start and end features, isn't it ?
EDIT 1 :
here is my try after PCamargo first answer :
def retrieve_feature_from_xy(geom,point,layerList):
for layer in layerList:
index = QgsSpatialIndex()
iter = layer.getFeatures()
for feat in iter:
index.insertFeature(feat)
ids = index.intersects(geom.boundingBox())
request = QgsFeatureRequest()
request.setFilterFids(ids)
iter = layer.getFeatures(request)
for feat in iter:
geom2 = feat.geometry().asPoint()
if geom2.x() == point.x() and geom2.y() == point.y():
return feat
EDIT 2 :
Here is my try after PCamargo second comment :
def retrieve_feature_from_xy2(geom,point,layerList):
allfeatures = {}
indexes=[]
ids=[]
for layer in layerList:
index = QgsSpatialIndex()
iter = layer.getFeatures()
for feat in iter:
index.insertFeature(feat)
allfeatures[feat.id()]=feat
indexes.append(index)
for index in indexes:
intersect_ids = index.intersects(geom.boundingBox())
ids.append(intersect_ids)
for id in ids:
for i in id:
feat=allfeatures[i]
geom2=feat.geometry().asPoint()
if geom2.x() == point.x() and geom2.y() == point.y():
return feat
EDIT 3
Here is my try after PCamargo third comment :
def retrieve_feature_from_xy3(geom,point,layerList):
allfeatures = {}
indexes=[]
ids=[]
indexDict = {}
intersectsIdsDict = {}
for layer in layerList:
index = QgsSpatialIndex()
iter = layer.getFeatures()
for feat in iter:
index.insertFeature(feat)
allfeatures[layer,feat.id()]=feat
indexes.append(index)
indexDict[layer]=index
for layer, index in indexDict.items():
intersectsIds = index.intersects(geom.boundingBox())
intersectsIdsDict[layer]=intersectsIds
for layer, index in intersectsIdsDict.items():
for id in index:
feat = allfeatures[layer,id]
geom2=feat.geometry().asPoint()
if geom2.x() == point.x() and geom2.y() == point.y():
return feat
Chris,
You can definitely improve the look for similar coordinates (Third part of the code).
Instead of looping through all features in each layer, create a spatial index (https://docs.qgis.org/2.2/en/docs/pyqgis_developer_cookbook/vector.html#using-spatial-index) for each link and use nearestNeighbor.
It would be something like this:
#You only need to create these indices once
indexes=[]
for layer in layerlist:
index = QgsSpatialIndex()
for feat in layer:
index.insertFeature(feat)
indexes.append(index)
Now that we have the indexes, we can use faster geographic search.
geom = feature.geometry ()
for index in indexes:
intersect_ids = index.intersects(geom.boundingBox())
intersect_ids is a smaller list of features that are candidates to be equivalent, so you can compare only these features with the feature you selected.
You need to organize this a bit more, but that is the idea