I'm trying to use Python to parse mxnet params into plain text. The code looks like the below. But the parsing result is not plain string, but some encoded text looks like this, "... \xaa>\x0f\xed\x8e>\xaf!\x8f>g ..." Could anybody give me some tips on it? Thanks a lot!
...
param_file = 'resnet-50-0000.params'
with open(param_file, 'rb') as f:
net_params = f.read()
...
The parameters are binary files. If you want to read them as plain text you need to decode them first as a dictionary of parameter_name->NDArray, that you can convert them to numpy. From numpy you can convert it to a list and then process it as a list (of lists) of scalar.
import mxnet as mx
params = mx.nd.load('resnet-50-0000.params')
for k, param in params.items():
print(k)
print(param.asnumpy().tolist())
Related
Suppose there exists a numpy array, data. I am trying to do the equivalent of the following
cv2.imwrite(filename, data)
with open(filename, 'rb') as fp:
data_compressed = filename.read()
without having to write to a file. Is there a way to convert numpy array to its equivalent PNG/JPG... representation without having to write to a file and read it as binary?
As Miki pointed out, imencode(...) is the solution.
I try to implement this code, using colab:
x = np.load('data_sample.npy',allow_pickle=True)
stacked_x = np.concatenate([x,x,x],1)
stacked_x.shape
but my sample was an image in jpg format, so I wasn't able to convert these images to .npy to use it.
actually, I tried this code:
import numpy as np
array = np.asarray('image.JPG')
print(array.tobytes())
print(array)
x = np.load(array,allow_pickle=True)
stacked_x = np.concatenate([x,x,x],1)
stacked_x.shape
print(x)
but I got:
and when I use this code :
x = np.load(array.tobytes(),allow_pickle=True)
I got:
so, any suggestion to solve this, precisely to convert .jpg to .npy?
Look at what your code does:
In [57]: np.asarray('image.JPG')
Out[57]: array('image.JPG', dtype='<U9')
In [58]: np.asarray('image.JPG').tobytes()
Out[58]: b'i\x00\x00\x00m\x00\x00\x00a\x00\x00\x00g\x00\x00\x00e\x00\x00\x00.\x00\x00\x00J\x00\x00\x00P\x00\x00\x00G\x00\x00\x00'
That's just playing with the string. It's not doing anything with a file named "image.JPG". You need to use some sort of image processing package to first load the file, converting it from the compressed format to a 3d array.
np.load is used to load a file that was created by np.save('file.npy'). It doesn't make sense to give that array variable as the file name. It won't load a jpg file.
np.load('file.npy',allow_pickle=True)
I have this code:
import cv2
im = cv2.imread("0.jpg")
print(len(im.tobytes()))
fp = open("0.jpg", 'rb')
imb = fp.read()
print(len(imb))
They are different! Now a function take 'imb' format as input. But I just have 'im'. I must use the cv2.imwrite to the disk and then use fp.read()?
Is there a faster way?
I use the mxnet image imdecode. imdecode take 'imb' as input. But im is what I get. How to pass the 'im' to the mx.img.imdecode?
https://github.com/apache/incubator-mxnet/issues/13545
I'm not familiar with mxnet framework. But according to the code you gave, I guess you want image data in binary format.
Take a look at OpenCV cv.imencode() API.
import cv2
im = cv2.imread("0.jpg")
img_encode = cv2.imencode('.jpg', im)[1]
I think the variable img_encode is might what you want.
Or try to use cv2.imwrite in more intuitive way like Memory Filesystem in "pyfilesystem" module.[link]
I'm trying to use the imageai python library, and more particularly this function:
detector.detectObjectsFromImage()
The doc says it should be used with a Numpy array of file stream of any image.
https://imageai.readthedocs.io/en/latest/detection/index.html
When I pass it a Numpy array, like this:
detections = detector.detectObjectsFromImage(input_image=anumpyarray,input_type = "array")
I get the error:
detections =
detector.detectObjectsFromImage(input_image=anumpyarray,input_type =
"array") File
"/usr/local/lib/python3.6/site-packages/imageai/Detection/init.py",
line 517, in detectObjectsFromImage raise ValueError("Ensure you
specified correct input image, input type, output type and/or output
image path ") ValueError: Ensure you specified correct input image,
input type, output type and/or output image path
Is it because a Numpy array and a Numpy array of a stream of an image are different things?
I know it's old, but for anyone who needs help:
Try to set 2 additional params:
minimum_percentage_probability=0, output_type='array'
For more info, go into imageai\Detection\__init__.py -> detectObjectsFromImage
I want to use tf.data.TextLineDataset() to read Chinese sentences, then use the map() function to divide into the single word, but tf.split doesn't work for Chinese.
I also hope someone can help us kindly with the issue.
It is my current solution:
read Chinese sentence from the file with Utf-8 coding format.
tokenize the sentences with some tool like jieba.
construct the vocab table.
convert source/target sentence according to vocab table.
convert to the dataset using from_tensor_slices.
get iterator from the dataset.
do other things.
if using TextLineDataset to load chinese sentences directlly, the content of dataset is something strange , displayed with byte flow.
maybe we can consider every byte as one character in english kind of language.
can anyone confirm with this or has any other suggestion, plz?
The above answer is one common option when handling non-English style language like Chinese, Korean, Japanese, etc.
You can also use the code below.
BTW, as you know, TextLineDataSet will read text content as a byte string.
So if we want to handle Chinese, we need to first decode it to unicode.
Unfortunately, there is no such option in tensorflow.
We need to choose other method like py_funct to do this.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import tensorflow as tf
def preprocess_func(x):
ret= "*".join(x.decode('utf-8'))
return ret
str = tf.py_func(
preprocess_func,
[tf.constant(u"我爱，南京")],
tf.string)
with tf.Session() as sess:
value = sess.run(str)
print(value.decode('utf-8'))
output: 我*爱*，*南*京