I want to implement a new operation similar to matrix multiplication in tensorflow. For example, If there are two matrices input, W and C = Op(input,W), then, C[i,j] = tf.math.reduce_prod(input[i,:] - W[:,j]). For implementing this operation I am planning to use two "for-loops" iterating over rows & columns, however, I believe that the for-loop implementation will decrease the training performance. I tried looking into the implementation of tf.lingalg.matmul() but the function seems to call C++ binaries which are not editable. Is there any way to speed up this computation? The following is the function I have written to implement Op(input,w):
def difference(input,w):
result = tf.Variable(tf.zeros((input.shape[0],w.shape[1]),dtype=tf.float16))
for i in tf.range(0,input.shape[0]):
for j in tf.range(0,w.shape[1]):
result[i,j].assign(tf.math.reduce_prod(input[i,:] - W[:,j]))
return result
Related
I need to translate this code to pytorch. The code given below use np.vectorize. I am looking for a pytorch equivalent for this.
class SimplexPotentialProjection(object):
def __init__(self, potential, inversePotential, strong_convexity_const, precision = 1e-10):
self.inversePotential = inversePotential
self.gradPsi = np.vectorize(potential)
self.gradPsiInverse = np.vectorize(inversePotential)
self.precision = precision
self.strong_convexity_const = strong_convexity_const
The doc for numpy.vectorize clearly states that:
The vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop.
Therefore, in order to convert your numpy code to pytorch you'll simply need apply potential and inversePotential in a loop over their tensor arguments.
However, that might be very inefficient. You would better re-implement your functions to act "natively" in a vectorized manner on tensors.
I am stuck with a problem when I need to compute an integration and run the loop for thousands of times.
The prototype of this function would be:
func_integrated = lambda t: scipy.integrate.quad(func, t, 1)
but then if I want to feed func_integrated into an array, that means I'll have to run this integration thousands of time, which is not very efficient. After all, the series from t to t-dt is just approximately func(t) *dt.
My guess would be to write something like this:
def function_integration(func, arr):
arr = np.flip(arr, 0) # flip to count form back to front
out_arr = np.array([0])
for i in range(1,len(arr)):
increment = arr[i-1] -arr[i]
out_arr = np.hstack((out_arr, increment * func(arr[i-1])))
return np.cumsum(out_arr)
However this is not generic, and still needs looping. I was wondering if there is any scipy or numpy way to handle this more efficiently.
P.S. I am aware of the sympy approach. However that relies heavily on built-in functions, so the method can hardly be generally adapted.
y = []
for i in range(9999):
x = ... # build/get sub graph
y.append(x)
y = tf.stack(x)
I am trying to build a Tensorflow Graph that contains multiple separate sub-graph/inputs (independently each other) that I am using a python for-loop to achieve this. (assuming the for-loop can not achieve by tensor operation).
I figure out this part is really slow and I think it is because the python for-loop is not in parallel (process them one by one). I wonder is there solution I can make it parallel (because each sub-graph/input is independent each other and separate)? I wonder change to tf.while_loop can be helpful or any other suggestions?
I have written a custom data iterator using mx.io.DataIter class. What's the easiest way to use this data iterator with Gluon interface?
I went through the documentation and couldn't find an easy way to do so. One of my idea was to use it as iterator and get data adn label from each batch as follows.
for e in range(epochs):
train_iter.reset()
for batch_data in train_iter:
data = nd.concatenate(([d for d in batch_data.data]))
label = nd.concatenate(([l for l in batch_data.label]))
with autograd.record():
output = net(data)
loss = softmax_cross_entropy(output, label)
loss.backward()
trainer.step(batch_size)
print(nd.mean(loss).asscalar())
But this may not be optimal as I need to concatenate per batch.
What's the optimal way to achieve this? i.e. is there a systematic
way to write a simple custom iterator for gluon?
How do I add context information in above cases?
I think your approach works. Basically you can get data from batch_data.data and label from batch_data.label and feed them into the network.
I'm not sure why you need to concat the data and labels - maybe it's to do with your network definition.
If you ever need to split the data and train on multiple GPUs, you can use the gluon.utils.split_and_load function to do that.
This question and answer demonstrate that when feature selection is performed using one of scikit-learn's dedicated feature selection routines, then the names of the selected features can be retrieved as follows:
np.asarray(vectorizer.get_feature_names())[featureSelector.get_support()]
For example, in the above code, featureSelector might be an instance of sklearn.feature_selection.SelectKBest or sklearn.feature_selection.SelectPercentile, since these classes implement the get_support method which returns a boolean mask or integer indices of the selected features.
When one performs feature selection via linear models penalized with the L1 norm, it's unclear how to accomplish this. sklearn.svm.LinearSVC has no get_support method and the documentation doesn't make clear how to retrieve the feature indices after using its transform method to eliminate features from a collection of samples. Am I missing something here?
For sparse estimators you can generally find the support by checking where the non-zero entries are in the coefficients vector (provided the coefficients vector exists, which is the case for e.g. linear models)
support = np.flatnonzero(estimator.coef_)
For your LinearSVC with l1 penalty it would accordingly be
from sklearn.svm import LinearSVC
svc = LinearSVC(C=1., penalty='l1', dual=False)
svc.fit(X, y)
selected_feature_names = np.asarray(vectorizer.get_feature_names())[np.flatnonzero(svc.coef_)]
I've been using sklearn 15.2, and according to LinearSVC documentation , coef_ is an array, shape = [n_features] if n_classes == 2 else [n_classes, n_features].
So first, np.flatnonzero doesn't work for multi-class. You'll have index out of range error. Second, it should be np.where(svc.coef_ != 0)[1] instead of np.where(svc.coef_ != 0)[0] . 0 is index of classes, not features. I ended up with using np.asarray(vectorizer.get_feature_names())[list(set(np.where(svc.coef_ != 0)[1]))]