How to extract raw IQ data from Gnuradio meta file sink? - gnuradio

Question

System:debian 11,Gnuradio 3.8.10,python 3.8.10
I try to use meta file sink block to record the raw IQ data with meta data.
grc file as below:
options:
parameters:
author: ''
category: '[GRC Hier Blocks]'
cmake_opt: ''
comment: ''
copyright: ''
description: ''
gen_cmake: 'On'
gen_linking: dynamic
generate_options: qt_gui
hier_block_src_path: '.:'
id: meta_sink
max_nouts: '0'
output_language: python
placement: (0,0)
qt_qss_theme: ''
realtime_scheduling: ''
run: 'True'
run_command: '{python} -u {filename}'
run_options: prompt
sizing_mode: fixed
thread_safe_setters: ''
title: Not titled yet
window_size: ''
states:
bus_sink: false
bus_source: false
bus_structure: null
coordinate: [8, 8]
rotation: 0
state: enabled
blocks:
- name: samp_rate
id: variable_qtgui_range
parameters:
comment: ''
gui_hint: ''
label: ''
min_len: '200'
orient: Qt.Horizontal
rangeType: float
start: '0'
step: '1'
stop: 128e3
value: 32e3
widget: counter_slider
states:
bus_sink: false
bus_source: false
bus_structure: null
coordinate: [384, 60.0]
rotation: 0
state: true
- name: strobe_value
id: variable_qtgui_chooser
parameters:
comment: ''
gui_hint: ''
label: ''
label0: str=0
label1: str=1
label2: ''
label3: ''
label4: ''
labels: '[]'
num_opts: '2'
option0: '0'
option1: '1'
option2: '2'
option3: '3'
option4: '4'
options: '[0, 1, 2]'
orient: Qt.QVBoxLayout
type: int
value: '0'
widget: combo_box
states:
bus_sink: false
bus_source: false
bus_structure: null
coordinate: [616, 20.0]
rotation: 0
state: true
- name: analog_sig_source_x_0
id: analog_sig_source_x
parameters:
affinity: ''
alias: ''
amp: '1'
comment: ''
freq: '100'
maxoutbuf: '0'
minoutbuf: '0'
offset: '0'
phase: '0'
samp_rate: samp_rate
type: complex
waveform: analog.GR_SAW_WAVE
states:
bus_sink: false
bus_source: false
bus_structure: null
coordinate: [80, 140.0]
rotation: 0
state: enabled
- name: blocks_file_meta_sink_0
id: blocks_file_meta_sink
parameters:
affinity: ''
alias: ''
comment: ''
detached: 'False'
extra_dict: pmt.make_dict()
file: /home/laozi/meta.bin
max_seg_size: '1000000'
rel_rate: '1'
samp_rate: samp_rate
type: complex
unbuffered: 'False'
vlen: '1'
states:
bus_sink: false
bus_source: false
bus_structure: null
coordinate: [632, 220.0]
rotation: 0
state: enabled
- name: blocks_throttle_0
id: blocks_throttle
parameters:
affinity: ''
alias: ''
comment: ''
ignoretag: 'True'
maxoutbuf: '0'
minoutbuf: '0'
samples_per_second: samp_rate
type: complex
vlen: '1'
states:
bus_sink: false
bus_source: false
bus_structure: null
coordinate: [448, 244.0]
rotation: 0
state: enabled
- name: import_0
id: import
parameters:
alias: ''
comment: ''
imports: import pmt
states:
bus_sink: false
bus_source: false
bus_structure: null
coordinate: [1224, 320.0]
rotation: 0
state: true
connections:
- [analog_sig_source_x_0, '0', blocks_throttle_0, '0']
- [blocks_throttle_0, '0', blocks_file_meta_sink_0, '0']
metadata:
file_format: 1
Then,use python script to extract the raw IQ data for further process:
from gnuradio import gr,blocks
import pmt
import sys
from gnuradio.blocks import parse_file_metadata
import numpy as np
filename ='./meta.bin'
max_data_segments_to_read = 3
print_output = True
fh = open(filename, "rb")
for ii in range(max_data_segments_to_read):
header_str = fh.read(parse_file_metadata.HEADER_LENGTH)
header = pmt.deserialize_str(header_str)
print(f"\n===Data segment {ii} ===")
header_info = parse_file_metadata.parse_header(header, print_output)
if(header_info["extra_len"] > 0):
extra_str = fh.read(header_info["extra_len"])
if(len(extra_str) != 0):
extra = pmt.deserialize_str(extra_str)
extra_info = parse_file_metadata.parse_extra_dict(extra, header_info, print_output)
data=np.fromfile(file=fh, dtype=np.float32, count=int(header_info['nitems']), sep='', offset=0)
print(f"{len(data)} data elements read")
fh.close()
But I got failed as below:
===Data segment 0 ===
Version Number: 0
Sample Rate: 32000.00 sps
Seconds: 0.000000
Item size: 8
Data Type: float (5)
Complex? True
Header Length: 150 bytes
Extra Length: 1
Extra Header? True
Size of Data: 8000000 bytes
1000000.0 items
1000000 data elements read
Traceback (most recent call last):
File "/home/laozi/a.py", line 13, in <module>
header = pmt.deserialize_str(header_str)
File "/usr/lib/python3/dist-packages/pmt/pmt_swig.py", line 2898, in deserialize_str
return _deserialize_str_u8(tuple(x for x in pmt_str))
File "/usr/lib/python3/dist-packages/pmt/pmt_swig.py", line 2871, in _deserialize_str_u8
return _pmt_swig._deserialize_str_u8(py_str)
RuntimeError: pmt::deserialize: malformed input stream, tag value = : 91
Where is the problem?

Answer 1

Item size: 8
Size of Data: 8000000 bytes
1000000.0 items
One possible explanation is that you are not reading all the data. The metadata indicates that the size of one item is 8 bytes (two float32 numbers - one for I and one for Q) and there are 1000000 items (8000000 bytes).
data=np.fromfile(file=fh, dtype=np.float32, count=int(header_info['nitems']), sep='', offset=0)
You've specified dtype to be np.float32 (that is 4 bytes) and count is the number of items (1000000) so only 4000000 bytes are being read.
Try reading twice as much - count=2*int(header_info['nitems'])