GNU Radio File Format for the recorded samples - gnuradio

Do you know the format in which GNU Radio ( File Sink in GNU Radio Companion) stores the samples in the Binary File?
I need to read these samples in Matlab, but the problem is the file is too big to be read in Matlab.
I am writing the program in C++ to read this binary file.

The file sink is just a dump of the data stream. If the data stream content was simple bytes then the content of the file is straightforward. If the data stream contained complex numbers then the file will contain a list of complex numbers where each complex number is given by two floats and each float by (usually) 4 bytes.
See the files gnuradio/gnuradio-core/src/lib/io/gr_file_sink.cc and gr_file_source.cc for the implementations of the gnuradio file reading and writing blocks.
You could also use python and gnuradio to convert the files into some other format.
from gnuradio import gr
# Assuming the data stream was complex numbers.
src = gr.file_source(gr.sizeof_gr_complex, "the_file_name")
snk = gr.vector_sink_c()
tb = gr.top_block()
tb.connect(src, snk)
tb.run()
# The complex numbers are then accessible as a python list.
data = snk.data()

Ben's answer still stands – but it's from a time long past (the module organization points at GNU Radio 3.6, I think). Organizationally, things are different now; data-wise, the File Sink remained the same.
GNU Radio now has relatively much block documentation in their wiki. In particular, the File Sink documentation page has a section on Handling File Sink data; not to overquote that:
// This is C++17
#include <algorithm>
#include <cmath>
#include <complex>
#include <cstddef>
#include <filesystem>
#include <fstream>
#include <string_view>
#include <vector>
#include <fmt/format.h>
#include <fmt/ranges.h>
using sample_t = std::complex<float>;
using power_t = float;
constexpr std::size_t read_block_size = 1 << 16;
int main(int argc, char *argv[]) {
// expect exactly one argument, a file name
if (argc != 2) {
fmt::print(stderr, "Usage: {} FILE_NAME", argv[0]);
return -1;
}
// just for convenience; we could as well just use `argv[1]` throughout the
// code
std::string_view filename(argv[1]);
// check whether file exists
if (!std::filesystem::exists(filename.data())) {
fmt::print(stderr, "file '{:s}' not found\n", filename);
return -2;
}
// calculate how many samples to read
auto file_size = std::filesystem::file_size(std::filesystem::path(filename));
auto samples_to_read = file_size / sizeof(sample_t);
// construct and reserve container for resulting powers
std::vector<power_t> powers;
powers.reserve(samples_to_read);
std::ifstream input_file(filename.data(), std::ios_base::binary);
if (!input_file) {
fmt::print(stderr, "error opening '{:s}'\n", filename);
return -3;
}
// construct and reserve container for read samples
// if read_block_size == 0, then read the whole file at once
std::vector<sample_t> samples;
if (read_block_size)
samples.resize(read_block_size);
else
samples.resize(samples_to_read);
fmt::print(stderr, "Reading {:d} samples…\n", samples_to_read);
while (samples_to_read) {
auto read_now = std::min(samples_to_read, samples.size());
input_file.read(reinterpret_cast<char *>(samples.data()),
read_now * sizeof(sample_t));
for (size_t idx = 0; idx < read_now; ++idx) {
auto magnitude = std::abs(samples[idx]);
powers.push_back(magnitude * magnitude);
}
samples_to_read -= read_now;
}
// we're not actually doing anything with the data. Let's print it!
fmt::print("Power\n{}\n", fmt::join(powers, "\n"));
}

Related

PCL print current FPS

I am using PCL library to visualize a single .ply model. How can I print the Frames per Second (FPS) number shown on the bottom left of the window?
Here is my simple code. We only need to add something like cout<<print(current_fps);
#include <iostream>
//#include <unistd.h>
#include <pcl/io/pcd_io.h>
#include <pcl/io/ply_io.h>
#include <pcl/point_cloud.h>
#include <pcl/console/parse.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>
// Main function
int main(int argc, char **argv)
{
// Fetch point cloud filename in arguments | Works with PLY files
std::vector<int> filenames;
filenames = pcl::console::parse_file_extension_argument(argc, argv, ".ply");
// Load file | Works with PLY files
pcl::PointCloud<pcl::PointXYZRGB>::Ptr source_cloud (new pcl::PointCloud<pcl::PointXYZRGB>());
pcl::io::loadPLYFile(argv[filenames[0]], *source_cloud);
// Visualization
printf("\n Point cloud colors :\n"
" \t white \t = \t original point cloud \n");
pcl::visualization::PCLVisualizer viewer(" Point Cloud Datsets Visualizer");
viewer.setBackgroundColor(0.05, 0.05, 0.05, 0); // Set background to a dark grey
// Define R,G,B colors for the point cloud
pcl::visualization::PointCloudColorHandlerRGBField<pcl::PointXYZRGB> rgb(source_cloud);
// We add the point cloud to the viewer and pass the color handler
viewer.addPointCloud(source_cloud, rgb, "original_cloud");
viewer.setPointCloudRenderingProperties(pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 2, "original_cloud");
//******************************
std::vector<Point, Eigen::aligned_allocator<Point>> points = source_cloud->points;
//******************************
while (!viewer.wasStopped()) { // Display the visualizer until the 'q' key is pressed
viewer.spinOnce();
}
return 0;
} // End main()
You can use getFPS(). The issue was fixed in
https://github.com/PointCloudLibrary/pcl/pull/1974
Then you can do something like
std::cout<<viewer.getFPS();
Which will give you a floating point value with the current rendering framerate.

How to load a graph with tensorflow.so and c_api.h in c++ language?

I am not able to find any examples about how to load a graph with tensorflow.so and c_api.h in C++. I read the c_api.h, however the ReadBinaryProto function was not in it. How can I load a graph without the ReadBinaryProto function?
If you're using C++, you might want to use the C++ API instead. The label image example would probably be a good sample to help you start.
If you really want to use just the C API, use TF_GraphImportGraphDef to load a graph. Note that the C API isn't particularly convenient to use (it is intended to build bindings in other languages such as Go, Java, Rust, Haskell etc.) For example:
#include <stdio.h>
#include <stdlib.h>
#include <tensorflow/c/c_api.h>
TF_Buffer* read_file(const char* file);
void free_buffer(void* data, size_t length) {
free(data);
}
int main() {
// Graph definition from unzipped https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip
// which is used in the Go, Java and Android examples
TF_Buffer* graph_def = read_file("tensorflow_inception_graph.pb");
TF_Graph* graph = TF_NewGraph();
// Import graph_def into graph
TF_Status* status = TF_NewStatus();
TF_ImportGraphDefOptions* opts = TF_NewImportGraphDefOptions();
TF_GraphImportGraphDef(graph, graph_def, opts, status);
TF_DeleteImportGraphDefOptions(opts);
if (TF_GetCode(status) != TF_OK) {
fprintf(stderr, "ERROR: Unable to import graph %s", TF_Message(status));
return 1;
}
fprintf(stdout, "Successfully imported graph");
TF_DeleteStatus(status);
TF_DeleteBuffer(graph_def);
// Use the graph
TF_DeleteGraph(graph);
return 0;
}
TF_Buffer* read_file(const char* file) {
FILE *f = fopen(file, "rb");
fseek(f, 0, SEEK_END);
long fsize = ftell(f);
fseek(f, 0, SEEK_SET); //same as rewind(f);
void* data = malloc(fsize);
fread(data, fsize, 1, f);
fclose(f);
TF_Buffer* buf = TF_NewBuffer();
buf->data = data;
buf->length = fsize;
buf->data_deallocator = free_buffer;
return buf;
}
The previous answer is your main option if you are wanting to use it outside of the TensorFlow project (and consequently not build with Bazel). You need to load it from the c_api.h with TF_GraphImportDef, I recommend training and doing testing in Python and then exporting the model/graph for use with C++/C Api when you have finished.

what's the storage format of protobuf data?

the .proto file:
package lm;
message helloworld
{
required int32 id = 1;
required string str = 2;
optional int32 opt = 3;
}
the writer.cc file:
#include <iostream>
#include <string>
#include "lm.helloworld.pb.h"
#include <fstream>
using namespace std;
int main()
{
lm::helloworld msg1;
msg1.set_id(101000);
msg1.set_str("helloworld,this is a protobuf writer");
fstream output("log", ios::out | ios::trunc | ios::binary);
string _data;
msg1.SerializeToString(&_data);
cout << _data << endl;
if(!msg1.SerializeToOstream(&output))
{
cerr << "Failed to write msg" << endl;
return -1;
}
return 0;
}
the reader.cc file:
#include <iostream>
#include <fstream>
#include <string>
#include "lm.helloworld.pb.h"
using namespace std;
void ListMsg(const lm::helloworld & msg)
{
cout << msg.id() << endl;
cout << msg.str() << endl;
}
int main(int argc, char* argv[])
{
lm::helloworld msg1;
{
fstream input("log", ios::in | ios::binary);
if (!msg1.ParseFromIstream(&input))
{
cerr << "Failed to parse address book." << endl;
return -1;
}
}
ListMsg(msg1);
return 0;
}
It's a simple reader and writer model using protobuf. but what's in the log is a readable string a typed in the write.cc file instead of "numeric format",why is that?
The encoding is described here.
Without an example of what comes out the other end, that is slightly hard to answer precisely, but there are two possible explanations of what you are seeing:
you have explicitly switched into TextFormat in your code; this is very unlikely - and indeed, the primary use of TextFormat is debugging etc
far more likely, you're just seeing the text data from your message in the binary; text is encoded as UTF-8, so if you open a protobuf file in a text editor, pieces of it will appear readable enough to display something in the file
The real question is: what are the actual bytes in your output file? If it is something like:
08-88-95-06-12-24-68-65-6C-6C-6F-77-6F-72-6C-64-2C-74-68-69-73-20-69-73-20-61-20-70-72-6F-74-6F-62-75-66-20-77-72-69-74-65-72
then that is the binary format; but note that most of that is simply the UTF-8 of the string "helloworld,this is a protobuf writer" - which dominates the message by sheer size:
68-65-6C-6C-6F-77-6F-72-6C-64-2C-74-68-69-73-20-69-73-20-61-20-70-72-6F-74-6F-62-75-66-20-77-72-69-74-65-72
So if you look in any text editor, it will appear as a few garbled characters at the start followed by the clearly legible helloworld,this is a protobuf writer.
The "binary" here is the bit at the start:
08-88-95-06-12-24
This is:
08: header: field 1, varint
88-95-06: the value (decimal) 101000 as a varint
12: header: field 2, length-prefixed
24: the value (decimal) 36 as a varint (the length of the string, in bytes)
The key points to note:
if your message is dominated by text, the yes, large parts of it will look human-readable even in binary form
look at the overheads; it to 6 bytes to encode the entire rest of the message, and 3 bytes of that was data (the 101000) - so only 3 bytes was actually lost as overhead; now compare and contrast to xml, json, etc to understand what protobuf is doing to help you

/proc/[pid]/cmdline file size

i'm trying to get the filesize of the cmdline file in proc/[pid]. For example porc/1/cmdline. The file is not empty, it contains "/sbin/init". But i get file_size = 0.
int main(int argc, char **argv) {
int file_size;
FILE *file_cmd;
file_cmd = fopen("/proc/1/cmdline", "r");
if(file_cmd == NULL) {
perror("proc/1/cmdline");
exit(1);
}else {
if(fseek(file_cmd, 0L, SEEK_END)!=0) {
perror("proc/1/cmdline");
exit(1);
}
file_size = ftell(file_cmd);
}
printf("fs: %d\n",file_size);
fclose(file_cmd);
}
Regards
That's normal. /proc files (most of them, there are a few exceptions) are generated by the kernel at the moment you read from them. That means it's impossible to know the size before reading from the file. Think of it as Quantum Mechanics on files. You won't get a state unless you read the information, but there's no guarantee that reading again will give you the same information twice ;-)
In other words, the EOF is only generated when you try to read it. It's not there before that, so there's no way a file size can be determined.
This is really just communication with the kernel disguised as file I/O.

Trying to parse OpenCV YAML ouput with yaml-cpp

I've got a series of OpenCv generated YAML files and would like to parse them with yaml-cpp
I'm doing okay on simple stuff, but the matrix representation is proving difficult.
# Center of table
tableCenter: !!opencv-matrix
rows: 1
cols: 2
dt: f
data: [ 240, 240]
This should map into the vector
240
240
with type float. My code looks like:
#include "yaml.h"
#include <fstream>
#include <string>
struct Matrix {
int x;
};
void operator >> (const YAML::Node& node, Matrix& matrix) {
unsigned rows;
node["rows"] >> rows;
}
int main()
{
std::ifstream fin("monsters.yaml");
YAML::Parser parser(fin);
YAML::Node doc;
Matrix m;
doc["tableCenter"] >> m;
return 0;
}
But I get
terminate called after throwing an instance of 'YAML::BadDereference'
what(): yaml-cpp: error at line 0, column 0: bad dereference
Abort trap
I searched around for some documentation for yaml-cpp, but there doesn't seem to be any, aside from a short introductory example on parsing and emitting. Unfortunately, neither of these two help in this particular circumstance.
As I understand, the !! indicate that this is a user-defined type, but I don't see with yaml-cpp how to parse that.
You have to tell yaml-cpp how to parse this type. Since C++ isn't dynamically typed, it can't detect what data type you want and create it from scratch - you have to tell it directly. Tagging a node is really only for yourself, not for the parser (it'll just faithfully store it for you).
I'm not really sure how an OpenCV matrix is stored, but if it's something like this:
class Matrix {
public:
Matrix(unsigned r, unsigned c, const std::vector<float>& d): rows(r), cols(c), data(d) { /* init */ }
Matrix(const Matrix&) { /* copy */ }
~Matrix() { /* delete */ }
Matrix& operator = (const Matrix&) { /* assign */ }
private:
unsigned rows, cols;
std::vector<float> data;
};
then you can write something like
void operator >> (const YAML::Node& node, Matrix& matrix) {
unsigned rows, cols;
std::vector<float> data;
node["rows"] >> rows;
node["cols"] >> cols;
node["data"] >> data;
matrix = Matrix(rows, cols, data);
}
Edit It appears that you're ok up until here; but you're missing the step where the parser loads the information into the YAML::Node. Instead, se it like:
std::ifstream fin("monsters.yaml");
YAML::Parser parser(fin);
YAML::Node doc;
parser.GetNextDocument(doc); // <-- this line was missing!
Matrix m;
doc["tableCenter"] >> m;
Note: I'm guessing dt: f means "data type is float". If that's the case, it'll really depend on how the Matrix class handles this. If you have a different class for each data type (or a templated class), you'll have to read that field first, and then choose which type to instantiate. (If you know it'll always be float, that'll make your life easier, of course.)