Unable to convert VGG-16 to IR - tensorflow

I have truncated version of vgg16 in .pb format. I am unable to convert to IR using OpenVino Model Optimizer getting following error:
[ ANALYSIS INFO ] It looks like there is IteratorGetNext as input
Run the Model Optimizer with:
--input "IteratorGetNext:0[-1 224 224 3]"
And replace all negative values with positive values
[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (): Graph contains 0 node after executing . It considered as error because resulting IR will be empty which is not usual
python3 /opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo_tf.py --input_model model.pb
With *.meta
python3 /opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo_tf.py --input_meta_graph model.meta --log_level DEBUG
[ 2020-06-11 10:59:34,182 ] [ DEBUG ] [ main:213 ] Placeholder shapes : None
'extensions.back.ScalarConstNormalize.RangeInputNormalize'>
| 310 | True | <class 'extensions.back.AvgPool.AvgPool'>
| 311 | True | <class 'extensions.back.ReverseInputChannels.ApplyReverseChannels'>
| 312 | True | <class 'extensions.back.split_normalizer.SplitNormalizer'>
| 313 | True | <class 'extensions.back.ParameterToPlaceholder.ParameterToInput'>
| 314 | True | <class 'extensions.back.GroupedConvWeightsNormalize.GroupedConvWeightsNormalize'>
| 315 | True | <class 'extensions.back.ConvolutionNormalizer.DeconvolutionNormalizer'>
| 316 | True | <class 'extensions.back.StridedSliceMasksNormalizer.StridedSliceMasksNormalizer'>
| 317 | True | <class 'extensions.back.ConvolutionNormalizer.ConvolutionWithGroupsResolver'>
| 318 | True | <class 'extensions.back.ReshapeMutation.ReshapeMutation'>
| 319 | True | <class 'extensions.back.ForceStrictPrecision.ForceStrictPrecision'>
| 320 | True | <class 'extensions.back.I64ToI32.I64ToI32'>
| 321 | True | <class 'extensions.back.ReshapeMutation.DisableReshapeMutationInTensorIterator'>
| 322 | True | <class 'extensions.back.ActivationsNormalizer.ActivationsNormalizer'>
| 323 | True | <class 'extensions.back.pass_separator.BackFinish'>
| 324 | False | <class 'extensions.back.SpecialNodesFinalization.RemoveConstOps'>
| 325 | False | <class 'extensions.back.SpecialNodesFinalization.CreateConstNodesReplacement'>
| 326 | True | <class 'extensions.back.kaldi_remove_memory_output.KaldiRemoveMemoryOutputBackReplacementPattern'>
| 327 | False | <class 'extensions.back.SpecialNodesFinalization.RemoveOutputOps'>
| 328 | True | <class 'extensions.back.blob_normalizer.BlobNormalizer'>
| 329 | False | <class 'extensions.middle.MulFakeQuantizeFuse.MulFakeQuantizeFuse'>
| 330 | False | <class 'extensions.middle.AddFakeQuantizeFuse.AddFakeQuantizeFuse'>
[ 2020-06-11 10:59:34,900 ] [ DEBUG ] [ class_registration:282 ] Run replacer <class 'extensions.load.tf.loader.TFLoader'>
[ INFO ] Restoring parameters from %s
[ WARNING ] From %s: %s (from %s) is deprecated and will be removed %s.
Instructions for updating:
%s
[ WARNING ] From %s: %s (from %s) is deprecated and will be removed %s.
Instructions for updating:
%s
[ FRAMEWORK ERROR ] Cannot load input model: Attempting to use uninitialized value metrics/accuracy/total
[[{{node _retval_metrics/accuracy/total_0_54}}]]
[ 2020-06-11 10:59:35,760 ] [ DEBUG ] [ main:328 ] Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value metrics/accuracy/total
[[{{node _retval_metrics/accuracy/total_0_54}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 220, in load_tf_graph_def
outputs)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/framework/graph_util_impl.py", line 330, in convert_variables_to_constants
returned_variables = sess.run(variable_names)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value metrics/accuracy/total
[[{{node _retval_metrics/accuracy/total_0_54}}]]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 288, in apply_transform
for_graph_and_each_sub_graph_recursively(graph, replacer.find_and_replace_pattern)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/middle/pattern_match.py", line 58, in for_graph_and_each_sub_graph_recursively
func(graph)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/extensions/load/loader.py", line 27, in find_and_replace_pattern
self.load(graph)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/extensions/load/tf/loader.py", line 58, in load
saved_model_tags=argv.saved_model_tags)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 231, in load_tf_graph_def
raise FrameworkError('Cannot load input model: {}', e) from e
mo.utils.error.FrameworkError: Cannot load input model: Attempting to use uninitialized value metrics/accuracy/total
[[{{node _retval_metrics/accuracy/total_0_54}}]]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/main.py", line 312, in main
ret_code = driver(argv)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/main.py", line 273, in driver
ret_res = emit_ir(prepare_ir(argv), argv)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/main.py", line 238, in prepare_ir
graph = unified_pipeline(argv)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/pipeline/unified.py", line 29, in unified_pipeline
class_registration.ClassType.BACK_REPLACER
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 334, in apply_replacements
apply_replacements_list(graph, replacers_order)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 324, in apply_replacements_list
num_transforms=len(replacers_order))
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/utils/logger.py", line 124, in wrapper
function(*args, **kwargs)
File "/opt/intel/openvino_2020.3.194/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 306, in apply_transform
raise FrameworkError('{}'.format(str(err))) from err
mo.utils.error.FrameworkError: Cannot load input model: Attempting to use uninitialized value metrics/accuracy/total
[[{{node _retval_metrics/accuracy/total_0_54}}]]

The problem is that models trained in TensorFlow have some shapes undefined. In your case, it looks like batch of the input is not defined. To fix it, please add an additional argument to the command line: -b 1. The option sets batch to 1. It should fix this particular issue.
After that, I guess, you may encounter other issues so I would leave the following link: Converting a TensorFlow Model.
There are some tips about how to convert TensorFlow model to IR.

Related

Replacing unique array of strings in a row using pyspark

I am trying the following code which replace an empty list with unique array of a column("apples_set") when the condition "all" is satisfied.
The column "apple_logic_string" is of type Array[String]
Data frame looks like this:
apples_patterns.show()
+--------------------+-----------------+
| apples_logic_string|apples_set |
+--------------------+-----------------+
| "234" |["43","54"] |
| "65" |["95"] |
| "all" |[] |
| "76" |["84","67"] |
+--------------------+-----------------+
The code:
unique_all_apples = set(apples_patterns.agg(F.flatten(F.collect_set('apples_set'))).head()[0]) # noqa
error_patterns = apples_patterns.withColumn('apples_set', F.when(F.col('apples_logic_string') == 'all',
unique_all_apples).otherwise(F.col('apples_set')))
The Error:
Traceback (most recent call last):
File "/myproject/datasets/apples_matching.py", line 24, in compute
apples_patterns = apples_patterns.withColumn('apples_set', F.when(F.col('apples_logic_string') == 'all',
File "/scratch/asset-install/1c9821b4f6adc95ac4d5f15ff109001b/miniconda38/lib/python3.8/site-packages/pyspark/sql/functions.py", line 1518, in when
jc = sc._jvm.functions.when(condition._jc, v)
File "/scratch/asset-install/1c9821b4f6adc95ac4d5f15ff109001b/miniconda38/lib/python3.8/site-packages/py4j/java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "/scratch/asset-install/1c9821b4f6adc95ac4d5f15ff109001b/miniconda38/lib/python3.8/site-packages/pyspark/sql/utils.py", line 111, in deco
return f(*a, **kw)
File "/scratch/asset-install/1c9821b4f6adc95ac4d5f15ff109001b/miniconda38/lib/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.sql.functions.when.
: java.lang.RuntimeException: Unsupported literal type class java.util.ArrayList [43,54,95,84,67]
You can use array function: array documentation
In your case you may use it like this:
F.array([F.lit(x) for x in unique_all_apples])
sample code
import pyspark.sql.functions as F
x = [("234", ["43", "54"]), ("65", ["95"]), ("all", []), ("76", ["84", "67"])]
apples_patterns = spark.createDataFrame(x, schema=["apples_logic_string", "apples_set"])
unique_all_apples = set(
apples_patterns.agg(F.flatten(F.collect_set("apples_set"))).head()[0]
)
error_patterns = apples_patterns.withColumn(
"apples_set",
F.when(
F.col("apples_logic_string") == "all",
F.array([F.lit(x) for x in unique_all_apples]),
).otherwise(F.col("apples_set")),
)
And the output:
+-------------------+--------------------+
|apples_logic_string| apples_set|
+-------------------+--------------------+
| 234| [43, 54]|
| 65| [95]|
| all|[54, 95, 43, 67, 84]|
| 76| [84, 67]|
+-------------------+--------------------+
The easiest solution is to create another dataframe with one row that contains all distinct apples_set using explode than collect_set, after that joined to the original dataframe:
import spark.implicits._
val data = Seq(
("234", Seq("43", "54")),
("65", Seq("95")),
("all", Seq()),
("76", Seq("84", "67"))
)
val df = spark.sparkContext.parallelize(data).toDF("apples_logic_string", "apples_set")
val allDf = df.select(explode(col("apples_set")).as("apples_set")).agg(collect_set("apples_set").as("all_apples_set"))
.withColumn("apples_logic_string", lit("all"))
df.join(broadcast(allDf), Seq("apples_logic_string"), "left")
.withColumn("apples_set", when(col("apples_logic_string").equalTo("all"), col("all_apples_set")).otherwise(col("apples_set")))
.drop("all_apples_set").show(false)
+-------------------+--------------------+
|apples_logic_string|apples_set |
+-------------------+--------------------+
|234 |[43, 54] |
|65 |[95] |
|all |[84, 95, 67, 54, 43]|
|76 |[84, 67] |
+-------------------+--------------------+

error in building model "ner_ontonotes_bert_mult" in DeepPavlov

I'm trying to build model "ner_ontonotes_bert_mult" via GoogleColab using example in documentation:
from deeppavlov import build_model, configs
ner_model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)
but get error:
TypeError: init() got an unexpected keyword argument 'num_tags'
p.s. if I try to load another model (e.g. "ner_rus_bert"), the error does not appear
Full error (* maybe the error is related to directoty /packages/deeppavlov/models/torch_bert/crf.py* ):
2022-12-17 13:08:23.235 INFO in 'deeppavlov.download'['download'] at line 138: Skipped http://files.deeppavlov.ai/v1/ner/ner_ontonotes_bert_mult_torch_crf.tar.gz download because of matching hashes
INFO:deeppavlov.download:Skipped http://files.deeppavlov.ai/v1/ner/ner_ontonotes_bert_mult_torch_crf.tar.gz download because of matching hashes
Some weights of the model checkpoint at bert-base-multilingual-cased were not used when initializing BertForTokenClassification: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForTokenClassification were not initialized from the model checkpoint at bert-base-multilingual-cased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2022-12-17 13:08:30.1 ERROR in 'deeppavlov.core.common.params'['params'] at line 108: Exception in <class 'deeppavlov.models.torch_bert.torch_transformers_sequence_tagger.TorchTransformersSequenceTagger'>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/core/common/params.py", line 102, in from_params
component = obj(**dict(config_params, **kwargs))
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 182, in __init__
super().__init__(optimizer=optimizer,
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/core/models/torch_model.py", line 98, in __init__
self.load()
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 295, in load
self.crf = CRF(self.n_classes).to(self.device)
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/crf.py", line 13, in __init__
super().__init__(num_tags=num_tags, batch_first=batch_first)
TypeError: __init__() got an unexpected keyword argument 'num_tags'
ERROR:deeppavlov.core.common.params:Exception in <class 'deeppavlov.models.torch_bert.torch_transformers_sequence_tagger.TorchTransformersSequenceTagger'>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/core/common/params.py", line 102, in from_params
component = obj(**dict(config_params, **kwargs))
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 182, in __init__
super().__init__(optimizer=optimizer,
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/core/models/torch_model.py", line 98, in __init__
self.load()
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py", line 295, in load
self.crf = CRF(self.n_classes).to(self.device)
File "/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/crf.py", line 13, in __init__
super().__init__(num_tags=num_tags, batch_first=batch_first)
TypeError: __init__() got an unexpected keyword argument 'num_tags'
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-75-5e156706e7e4> in <module>
1 # from deeppavlov import configs, build_model
2
----> 3 ner_model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True)
5 frames
/usr/local/lib/python3.8/dist-packages/deeppavlov/core/commands/infer.py in build_model(config, mode, load_trained, install, download)
53 .format(component_config.get('class_name', component_config.get('ref', 'UNKNOWN'))))
54
---> 55 component = from_params(component_config, mode=mode)
56
57 if 'id' in component_config:
/usr/local/lib/python3.8/dist-packages/deeppavlov/core/common/params.py in from_params(params, mode, **kwargs)
100 kwargs['mode'] = mode
101
--> 102 component = obj(**dict(config_params, **kwargs))
103 try:
104 _refs[config_params['id']] = component
/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py in __init__(self, n_tags, pretrained_bert, bert_config_file, attention_probs_keep_prob, hidden_keep_prob, optimizer, optimizer_parameters, learning_rate_drop_patience, learning_rate_drop_div, load_before_drop, clip_norm, min_learning_rate, use_crf, **kwargs)
180 self.use_crf = use_crf
181
--> 182 super().__init__(optimizer=optimizer,
183 optimizer_parameters=optimizer_parameters,
184 learning_rate_drop_patience=learning_rate_drop_patience,
/usr/local/lib/python3.8/dist-packages/deeppavlov/core/models/torch_model.py in __init__(self, device, optimizer, optimizer_parameters, lr_scheduler, lr_scheduler_parameters, learning_rate_drop_patience, learning_rate_drop_div, load_before_drop, min_learning_rate, *args, **kwargs)
96 self.opt = deepcopy(kwargs)
97
---> 98 self.load()
99 # we need to switch to eval mode here because by default it's in `train` mode.
100 # But in case of `interact/build_model` usage, we need to have model in eval mode.
/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/torch_transformers_sequence_tagger.py in load(self, fname)
293 self.model.to(self.device)
294 if self.use_crf:
--> 295 self.crf = CRF(self.n_classes).to(self.device)
296
297 self.optimizer = getattr(torch.optim, self.optimizer_name)(
/usr/local/lib/python3.8/dist-packages/deeppavlov/models/torch_bert/crf.py in __init__(self, num_tags, batch_first)
11
12 def __init__(self, num_tags: int, batch_first: bool = False) -> None:
---> 13 super().__init__(num_tags=num_tags, batch_first=batch_first)
14 nn.init.zeros_(self.transitions)
15 nn.init.zeros_(self.start_transitions)
TypeError: __init__() got an unexpected keyword argument 'num_tags'
Make sure that you are using the latest version of DeepPavlov by running:
!pip install deeppavlov
Then import all the required packages:
from deeppavlov import configs, build_model
Install the model's requirements and download the pretrained model:
ner_model = build_model(configs.ner.ner_ontonotes_bert_mult, download=True, install=True)
You can get more information about this model and many others in our recent Medium article.

trying to put python .pyc file to .py, "Unknown type 64 #" error occured

I'm trying to decompile a python .pyc file into a source code .py file with uncompyle6, but I'm getting the error "Unknown type 64 #".
Here is the error message:
Unknown type 64 #
Unknown type 0
Unknown type 0
Unknown type 0
Unknown type 64 #
Unknown type 0
Unknown type 0
Unknown type 0
Unknown type 15
Unknown type 24
Unknown type 125 }
Unknown type 2
Unknown type 124 |
Unknown type 2
Unknown type 32
Unknown type 1
Unknown type 87 W
Traceback (most recent call last):
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\load.py", line 304, in load_module_from_file_object
co = xdis.unmarshal.load_code(fp, magic_int, code_objects)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 565, in load_code
return um_gen.load()
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 179, in load
return self.r_object()
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 222, in r_object
return unmarshal_func(save_ref, bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 500, in t_code
co_cellvars = self.r_object(bytes_for_s=bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 222, in r_object
return unmarshal_func(save_ref, bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 381, in t_small_tuple
ret += (self.r_object(bytes_for_s=bytes_for_s),)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 222, in r_object
return unmarshal_func(save_ref, bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 500, in t_code
co_cellvars = self.r_object(bytes_for_s=bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 222, in r_object
return unmarshal_func(save_ref, bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 381, in t_small_tuple
ret += (self.r_object(bytes_for_s=bytes_for_s),)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 222, in r_object
return unmarshal_func(save_ref, bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 486, in t_code
co_code = self.r_object(bytes_for_s=True)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 222, in r_object
return unmarshal_func(save_ref, bytes_for_s)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\unmarshal.py", line 520, in t_code
code = to_portable(
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\codetype\__init__.py", line 200, in to_portable
return codeType2Portable(code, version_triple)
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\codetype\__init__.py", line 43, in codeType2Portable
return Code3(
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\codetype\code30.py", line 80, in __init__
self.check()
File "c:\users\angela\appdata\local\programs\python\python39\lib\site-packages\xdis\codetype\code13.py", line 87, in check
assert type(val) in fieldtype, "%s should be one of the types %s; is type %s" % (field, fieldtype, type(val))
AssertionError: co_code should be one of the types (<class 'str'>, <class 'bytes'>, <class 'list'>, <class 'tuple'>); is type <class 'NoneType'>
Ill-formed bytecode file fbs13.pyc
<class 'AssertionError'>; co_code should be one of the types (<class 'str'>, <class 'bytes'>, <class 'list'>, <class 'tuple'>); is type <class 'NoneType'>
I tried to change the magic number in the header of the .pyc file but no change.
After searching on the internet some people said it was because my operating system was 64bit.
Does anyone have any idea. thanks in advance.

the error message while running model_test.py for tensorflow deeplab

I have been trying to test the installation of deeplab by following this
# From tensorflow/models/research/
python deeplab/model_test.py
However, I got the following error message, in specific,
2018-04-25 10:54:23.488868: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at mkl_concat_op.cc:784 : Aborted: Operation received an exception:Status: 3, message: could not create a concat primitive descriptor, in file tensorflow/core/kernels/mkl_concat_op.cc:781
E...
======================================================================
ERROR: testForwardpassDeepLabv3plus (__main__.DeeplabModelTest)
----------------------------------------------------------------------
The complete traceback is as follows
2018-04-25 10:54:23.488868: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at mkl_concat_op.cc:784 : Aborted: Operation received an exception:Status: 3, message: could not create a concat primitive descriptor, in file tensorflow/core/kernels/mkl_concat_op.cc:781
E...
======================================================================
ERROR: testForwardpassDeepLabv3plus (__main__.DeeplabModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1312, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
status, run_metadata)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.AbortedError: Operation received an exception:Status: 3, message: could not create a concat primitive descriptor, in file tensorflow/core/kernels/mkl_concat_op.cc:781
[[Node: concat = _MklConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _kernel="MklOp", _device="/job:localhost/replica:0/task:0/device:CPU:0"](ResizeBilinear, aspp0/Relu, concat/axis, DMT/_283, aspp0/Relu:1, DMT/_284)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "deeplab/model_test.py", line 108, in testForwardpassDeepLabv3plus
outputs_to_scales_to_logits = sess.run(outputs_to_scales_to_logits)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1140, in _run
feed_dict_tensor, options, run_metadata)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
run_metadata)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AbortedError: Operation received an exception:Status: 3, message: could not create a concat primitive descriptor, in file tensorflow/core/kernels/mkl_concat_op.cc:781
[[Node: concat = _MklConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _kernel="MklOp", _device="/job:localhost/replica:0/task:0/device:CPU:0"](ResizeBilinear, aspp0/Relu, concat/axis, DMT/_283, aspp0/Relu:1, DMT/_284)]]
Caused by op 'concat', defined at:
File "deeplab/model_test.py", line 120, in <module>
tf.test.main()
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/platform/test.py", line 76, in main
return _googletest.main(argv)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/platform/googletest.py", line 99, in main
benchmark.benchmarks_main(true_main=main_wrapper)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/platform/benchmark.py", line 338, in benchmarks_main
true_main()
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/platform/googletest.py", line 98, in main_wrapper
return app.run(main=g_main, argv=args)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/platform/googletest.py", line 69, in g_main
return unittest_main(argv=argv)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/main.py", line 95, in __init__
self.runTests()
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/main.py", line 256, in runTests
self.result = testRunner.run(self.test)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/runner.py", line 176, in run
test(result)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/suite.py", line 122, in run
test(result)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/suite.py", line 122, in run
test(result)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/case.py", line 653, in __call__
return self.run(*args, **kwds)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/unittest/case.py", line 605, in run
testMethod()
File "deeplab/model_test.py", line 105, in testForwardpassDeepLabv3plus
image_pyramid=[1.0])
File "/data/dsp_emerging/ugwz/virtualE/deeplab/models/research/deeplab/model.py", line 296, in multi_scale_logits
fine_tune_batch_norm=fine_tune_batch_norm)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/models/research/deeplab/model.py", line 461, in _get_logits
fine_tune_batch_norm=fine_tune_batch_norm)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/models/research/deeplab/model.py", line 424, in _extract_features
concat_logits = tf.concat(branch_logits, 3)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1181, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 949, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3290, in create_op
op_def=op_def)
File "/data/dsp_emerging/ugwz/virtualE/deeplab/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1654, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
AbortedError (see above for traceback): Operation received an exception:Status: 3, message: could not create a concat primitive descriptor, in file tensorflow/core/kernels/mkl_concat_op.cc:781
[[Node: concat = _MklConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _kernel="MklOp", _device="/job:localhost/replica:0/task:0/device:CPU:0"](ResizeBilinear, aspp0/Relu, concat/axis, DMT/_283, aspp0/Relu:1, DMT/_284)]]
----------------------------------------------------------------------
Ran 5 tests in 23.571s
FAILED (errors=1)
Roll back to Tensorflow 1.6
This issue is still being addressed in versions 1.7 and above.
https://github.com/tensorflow/tensorflow/issues/17494
In Google Colab, in Runtime type Python2 or Python3, with GPU, I run without any error using commands:
!git clone https://github.com/tensorflow/models.git
%env PYTHONPATH=/env/python/:/content/models/research/:/content/models/research/slim
!python /content/models/research/deeplab/model_test.py

Apache Beam Dataflow: 'NoneType' object has no attribute 'parts'

I'm trying to write a pipeline to read a stream from pubsub and write it to bigquery using google cloud dataflow with apache beam.
I have this code:
import apache_beam as beam
from apache_beam.transforms.window import FixedWindows
topic = 'projects/???/topics/???'
table = '???.???'
gcs_path = "gs://???"
with beam.Pipeline(runner="DataflowRunner", argv=[
"--project", "???",
"--staging_location", ("%s/staging_location" % gcs_path),
"--temp_location", ("%s/temp" % gcs_path),
"--output", ("%s/output" % gcs_path)
]) as p:
(p
| 'winderow' >> beam.WindowInto(FixedWindows(60))
| 'hello' >> beam.io.gcp.pubsub.ReadStringsFromPubSub(topic)
| 'hello2' >> beam.io.Write(beam.io.gcp.bigquery.BigQuerySink(table))
)
p.run().wait_until_finish()
But I'm getting this error when running it:
No handlers could be found for logger "oauth2client.contrib.multistore_file"
ERROR:root:Error while visiting winderow
Traceback (most recent call last):
File ".\main.py", line 20, in <module>
p.run().wait_until_finish()
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\pipeline.py", line 339, in run
return self.runner.run(self)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\runners\dataflow\dataflow_runner.py", line 296, in run
super(DataflowRunner, self).run(pipeline)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\runners\runner.py", line 138, in run
pipeline.visit(RunVisitor(self))
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\pipeline.py", line 367, in visit
self._root_transform().visit(visitor, self, visited)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\pipeline.py", line 710, in visit
part.visit(visitor, pipeline, visited)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\pipeline.py", line 713, in visit
visitor.visit_transform(self)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\runners\runner.py", line 133, in visit_transform
self.runner.run_transform(transform_node)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\runners\runner.py", line 176, in run_transform
return m(transform_node)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\runners\dataflow\dataflow_runner.py", line 526, in run_ParDo
input_step = self._cache.get_pvalue(transform_node.inputs[0])
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\runners\runner.py", line 252, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
File "C:\ProgramData\Anaconda2\lib\site-packages\apache_beam\runners\runner.py", line 226, in _ensure_pvalue_has_real_producer
while real_producer.parts:
AttributeError: 'NoneType' object has no attribute 'parts'
Is this a problem with the code or configuration?
How can I get it working?
I don't have experience with windowed pipelines yet but for what I understand from the concept, the windows are supposed to be applied to your input data and not as a pipeline setup.
This being the case, your code probably should be:
with beam.Pipeline(runner="DataflowRunner", argv=[
"--project", "???",
"--staging_location", ("%s/staging_location" % gcs_path),
"--temp_location", ("%s/temp" % gcs_path),
"--output", ("%s/output" % gcs_path)
]) as p:
(p
| 'hello' >> beam.io.gcp.pubsub.ReadStringsFromPubSub(topic)
| 'winderow' >> beam.WindowInto(FixedWindows(60))
| 'hello2' >> beam.io.Write(beam.io.gcp.bigquery.BigQuerySink(table))
)
p.run().wait_until_finish()
The official repo have some samples on windowed operations as well.