numpy array advanced indexing in numba - numpy

I'm converting a function to numba (nopython mode). In this function I need to access some elements of a 2D array, whose row indices are contained in a list and the column is fixed.
In usual python I do:
array[list,column_index]
but when compiling in numba I receive the following error:
TypingError: Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (array(float64, 2d, C), (reflected list(int64), int64))
* parameterized
In definition 0:
All templates rejected with literals.
In definition 1:
All templates rejected without literals.
In definition 2:
All templates rejected with literals.
In definition 3:
All templates rejected without literals.
In definition 4:
All templates rejected with literals.
In definition 5:
All templates rejected without literals.
In definition 6:
TypeError: unsupported array index type reflected list(int64) in (reflected list(int64), int64)
raised from C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\typing\arraydecl.py:71
In definition 7:
TypeError: unsupported array index type reflected list(int64) in (reflected list(int64), int64)
raised from C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\typing\arraydecl.py:71
This error is usually caused by passing an argument of a type that is unsupported by the named function.
I already tried converting the list to an array as it would seem the correct way from https://numba.pydata.org/numba-doc/dev/reference/numpysupported.html
"A subset of advanced indexing is also supported: only one advanced index is allowed, and it has to be a one-dimensional array (it can be combined with an arbitrary number of basic indices as well)"
but coding:
array[np.array(list),column_index]
produces the following error:
Traceback (most recent call last):
File "<ipython-input-123-c5ca441bb6b7>", line 1, in <module>
tmp = L.numba_montecarlo_barab(SEED,w,R,T,mpx,0.5)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\dispatcher.py", line 367, in _compile_for_args
raise e
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\dispatcher.py", line 324, in _compile_for_args
return self.compile(tuple(argtypes))
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\dispatcher.py", line 655, in compile
cres = self._compiler.compile(args, return_type)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\dispatcher.py", line 82, in compile
pipeline_class=self.pipeline_class)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 926, in compile_extra
return pipeline.compile_extra(func)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 374, in compile_extra
return self._compile_bytecode()
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 857, in _compile_bytecode
return self._compile_core()
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 844, in _compile_core
res = pm.run(self.status)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler_lock.py", line 32, in _acquire_compile_lock
return func(*args, **kwargs)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 255, in run
raise patched_exception
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 246, in run
stage()
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 717, in stage_nopython_backend
self._backend(lowerfn, objectmode=False)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 666, in _backend
lowered = lowerfn()
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 653, in backend_nopython_mode
self.metadata)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\compiler.py", line 1050, in native_lowering_stage
lower.create_cpython_wrapper(flags.release_gil)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\lowering.py", line 272, in create_cpython_wrapper
release_gil=release_gil)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\targets\cpu.py", line 149, in create_cpython_wrapper
builder.build()
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\callwrapper.py", line 122, in build
self.build_wrapper(api, builder, closure, args, kws)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\callwrapper.py", line 155, in build_wrapper
val = cleanup_manager.add_arg(builder.load(obj), ty)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\callwrapper.py", line 58, in add_arg
cleanup_arg()
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\callwrapper.py", line 41, in cleanup_arg
self.api.reflect_native_value(ty, native.value, self.env_manager)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\pythonapi.py", line 1417, in reflect_native_value
impl(typ, val, c)
File "C:\ProgramData\Anaconda2\envs\Python 3.5\lib\site-packages\numba\targets\boxing.py", line 740, in reflect_list
raise TypeError(msg)
TypeError: Failed in nopython mode pipeline (step: nopython mode backend)
cannot reflect element of reflected container: reflected list(reflected list(reflected list(int64)))
also converting to a tuple didn't work, but I understand that tuple are not supported.
What is the correct way to use advanced indexing in numba nopython mode?

Related

pandas 1.3.3 to_feather giving ArrowMemoryError

I have a dataset of size around 270MB and I use the following to write to feather file:
df.reset_index().to_feather(feather_path)
This gives me an error :
File "C:\apps\Python\lib\site-packages\pandas\util\_decorators.py", line 207, in wrapper
return func(*args, **kwargs)
File "C:\apps\Python\lib\site-packages\pandas\core\frame.py", line 2519, in to_feather
to_feather(self, path, **kwargs)
File "C:\apps\Python\lib\site-packages\pandas\io\feather_format.py", line 87, in to_feather
feather.write_feather(df, handles.handle, **kwargs)
File "C:\apps\Python\lib\site-packages\pyarrow\feather.py", line 152, in write_feather
table = Table.from_pandas(df, preserve_index=False)
File "pyarrow\table.pxi", line 1553, in pyarrow.lib.Table.from_pandas
File "C:\apps\Python\lib\site-packages\pyarrow\pandas_compat.py", line 607, in dataframe_to_arrays
arrays[i] = maybe_fut.result()
File "C:\apps\Python\lib\concurrent\futures\_base.py", line 438, in result
return self.__get_result()
File "C:\apps\Python\lib\concurrent\futures\_base.py", line 390, in __get_result
raise self._exception
File "C:\apps\Python\lib\concurrent\futures\thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\apps\Python\lib\site-packages\pyarrow\pandas_compat.py", line 575, in convert_column
result = pa.array(col, type=type_, from_pandas=True, safe=safe)
File "pyarrow\array.pxi", line 302, in pyarrow.lib.array
File "pyarrow\array.pxi", line 83, in pyarrow.lib._ndarray_to_array
File "pyarrow\error.pxi", line 114, in pyarrow.lib.check_status
pyarrow.lib.ArrowMemoryError: realloc of size 3221225472 failed
Note : This works well in PyCharm. No issues writing the feather file.
But when the python program is called in a Windows batch file like:
call python "myprogram.py"
and when I schedule the batch file in a task using Task Scheduler it fails with above memory error.
PyArrow version is 5.0.0 if that helps.
Any ideas please?

Reading keys from an .npz file with multiple workers in pytorch dataloader?

I have an .npz file where I have stored a dictionary. The dictionary has some keys and the values are numpy arrays. I want to read the dictionary in my getitem() method of the dataloader. When I set the dataloader num_workers to 1, everything runs fine. But when I increase the num workers, it throws the following error when reading the data from that npz file:
Traceback (most recent call last):
File "scripts/train.py", line 235, in <module>
train(args)
File "scripts/train.py", line 186, in train
solver(args.epoch, args.verbose)
File "/local-scratch/codebase/cap/lib/solver.py", line 174, in __call__
self._feed(self.dataloader["train"], "train", epoch_id)
File "/local-scratch/codebase/cap/lib/solver.py", line 366, in _feed
for data_dict in dataloader:
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in __next__
return self._process_data(data)
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/_utils.py", line 369, in reraise
raise self.exc_type(msg)
zipfile.BadZipFile: Caught BadZipFile in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/local-scratch/codebase/cap/lib/dataset.py", line 947, in __getitem__
other_bbox_feat = self.box_features['{}-{}_{}.{}'.format(scene_id, target_object_id, ann_id, object_id)]
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/numpy/lib/npyio.py", line 255, in __getitem__
pickle_kwargs=self.pickle_kwargs)
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/numpy/lib/format.py", line 763, in read_array
data = _read_bytes(fp, read_size, "array data")
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/numpy/lib/format.py", line 892, in _read_bytes
r = fp.read(size - len(data))
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/zipfile.py", line 872, in read
data = self._read1(n)
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/zipfile.py", line 962, in _read1
self._update_crc(data)
File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/zipfile.py", line 890, in _update_crc
raise BadZipFile("Bad CRC-32 for file %r" % self.name)
zipfile.BadZipFile: Bad CRC-32 for file 'scene0519_00-13_1.0.npy'
As far as I know, pytorch dataloader uses multiprocessing to for data loading. Perhaps the issue is with multiprocessing and .npz files. I really appreciate any help.

Error while exciting the eval.py on TF object detection API

i'm trying to evaluate my model
using this command:
python eval.py --logtostderr --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config --checkpoint_dir=inference_graph --eval_dir=eval
and im getting this error
and I'm getting this error:
Traceback (most recent call last): File "eval.py", line 142, in tf.app.run() File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\tensorflow_core\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\absl\app.py", line 299, in run _run_main(main, args) File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\absl\app.py", line 250, in _run_main sys.exit(main(argv)) File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func return func(*args, **kwargs) File "eval.py", line 138, in main graph_hook_fn=graph_rewriter_fn) File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\legacy\evaluator.py", line 274, in evaluate evaluator_list = get_evaluators(eval_config, categories) File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\legacy\evaluator.py", line 166, in get_evaluators EVAL_METRICS_CLASS_DICTeval_metric_fn_key) File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\utils\object_detection_evaluation.py", line 470, in init use_weighted_mean_ap=False) File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\utils\object_detection_evaluation.py", line 194, in init self._build_metric_names() File "C:\Users\mosta\Anaconda3\envs\mat\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\utils\object_detection_evaluation.py", line 213, in _build_metric_names category_name = unicode(category_name, 'utf-8') NameError: name 'unicode' is not defined
Hi there!
Python 3 renamed the unicode type to str, the old str type has been replaced by bytes.
Knowing this it makes sense that we're getting errors as parts of the TF Object Detection API are deprecated (written using Python 2.x)
See here for more explanation on how to upgrade the code to be compatible with Python 3.
I hope this helps!

TypeError: loop of ufunc does not support argument 0 of type decimal.Decimal which has no callable arccos method

I'm trying to convert the Pyspark data frame into a pandas data frame.
meanwhile calculating long, lat values and updating them into a data frame.
def cos_max_longitude(radian_longitude, radian_latitude):
return radian_longitude + np.arcsin(np.sin(r) / np.arccos(radian_latitude))
Error Stack :
Traceback (most recent call last):
File "/usr/local/src/spark/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main
process()
File "/usr/local/src/spark/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/usr/local/src/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 345, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
File "/usr/local/src/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
for obj in iterator:
File "/usr/local/src/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 334, in _batched
for item in iterator:
File "<string>", line 1, in <lambda>
File "/usr/local/src/spark/python/lib/pyspark.zip/pyspark/worker.py", line 85, in <lambda>
return lambda *a: f(*a)
File "/usr/local/src/spark/python/lib/pyspark.zip/pyspark/util.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/ec2-user/test.py", line 1847, in cos_max_longitude
np.arcsin(np.sin(r) / np.arccos(radian_latitude))
TypeError: loop of ufunc does not support argument 0 of type decimal.Decimal which has no callable arccos method
Basically error is in np.arccos(radian_latitude)
My suggestion would be to check data type of radian_latitude also check data type of the value stored inside it
If its 'decimal.Decimal' then convert it to simple float datatype.
Refer below links :
https://docs.python.org/3/library/decimal.html
https://docs.scipy.org/doc/numpy/reference/ufuncs.html#optional-keyword-arguments
https://docs.scipy.org/doc/numpy/reference/generated/numpy.arccos.html

tensorflow tutorial mnist_with_summary throws TypeError

I am running the mnist_with_summary tutorial to see how the TensorBoard works. It throws a TypeError right away.
Extracting /tmp/data/train-images-idx3-ubyte.gz
Extracting /tmp/data/train-labels-idx1-ubyte.gz
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz
Traceback (most recent call last):
File "/Users/bruceho/workspace/TestTensorflow/mysrc/examples/tutorials/mnist/mnist_with_summaries.py", line 166, in <module>
tf.app.run()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "/Users/bruceho/workspace/TestTensorflow/mysrc/examples/tutorials/mnist/mnist_with_summaries.py", line 163, in main
train()
File "/Users/bruceho/workspace/TestTensorflow/mysrc/examples/tutorials/mnist/mnist_with_summaries.py", line 110, in train
y = nn_layer(dropped, 500, 10, 'layer2', act=tf.nn.softmax)
File "/Users/bruceho/workspace/TestTensorflow/mysrc/examples/tutorials/mnist/mnist_with_summaries.py", line 104, in nn_layer
activations = act(preactivate, 'activation')
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/ops/nn_ops.py", line 582, in softmax
return _softmax(logits, gen_nn_ops._softmax, dim, name)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/ops/nn_ops.py", line 542, in _softmax
logits = _swap_axis(logits, dim, math_ops.sub(input_rank, 1))
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/ops/nn_ops.py", line 518, in _swap_axis
0, [math_ops.range(dim_index), [last_index],
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/ops/math_ops.py", line 991, in range
return gen_math_ops._range(start, limit, delta, name=name)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1675, in _range
delta=delta, name=name)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/framework/op_def_library.py", line 490, in apply_op
preferred_dtype=default_dtype)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 657, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/framework/constant_op.py", line 180, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/framework/constant_op.py", line 163, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape))
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/framework/tensor_util.py", line 353, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tensorflow/python/framework/tensor_util.py", line 290, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).__name__))
TypeError: Expected int32, got 'activation' of type 'str' instead.
I tried running from inside eclipse and command line with the same results. Any one experience the same problem?
I think you must have modified the original code somehow. Your problem lies in this line:activations = act(preactivate, 'activation'). So if you check the api of tf.nn.softmax, you would find that the second argument represents dim instead of name. So to fix the problem, just change this line into:activations = act(preactivate, name='activation')
Besides, I don't know if you have changed
diff = tf.nn.softmax_cross_entropy_with_logits(y, y_)
If not, you probably have softmax the output twice.