How to load a graph with tensorflow.so and c_api.h in c++ language? - tensorflow

I am not able to find any examples about how to load a graph with tensorflow.so and c_api.h in C++. I read the c_api.h, however the ReadBinaryProto function was not in it. How can I load a graph without the ReadBinaryProto function?

If you're using C++, you might want to use the C++ API instead. The label image example would probably be a good sample to help you start.
If you really want to use just the C API, use TF_GraphImportGraphDef to load a graph. Note that the C API isn't particularly convenient to use (it is intended to build bindings in other languages such as Go, Java, Rust, Haskell etc.) For example:
#include <stdio.h>
#include <stdlib.h>
#include <tensorflow/c/c_api.h>
TF_Buffer* read_file(const char* file);
void free_buffer(void* data, size_t length) {
free(data);
}
int main() {
// Graph definition from unzipped https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip
// which is used in the Go, Java and Android examples
TF_Buffer* graph_def = read_file("tensorflow_inception_graph.pb");
TF_Graph* graph = TF_NewGraph();
// Import graph_def into graph
TF_Status* status = TF_NewStatus();
TF_ImportGraphDefOptions* opts = TF_NewImportGraphDefOptions();
TF_GraphImportGraphDef(graph, graph_def, opts, status);
TF_DeleteImportGraphDefOptions(opts);
if (TF_GetCode(status) != TF_OK) {
fprintf(stderr, "ERROR: Unable to import graph %s", TF_Message(status));
return 1;
}
fprintf(stdout, "Successfully imported graph");
TF_DeleteStatus(status);
TF_DeleteBuffer(graph_def);
// Use the graph
TF_DeleteGraph(graph);
return 0;
}
TF_Buffer* read_file(const char* file) {
FILE *f = fopen(file, "rb");
fseek(f, 0, SEEK_END);
long fsize = ftell(f);
fseek(f, 0, SEEK_SET); //same as rewind(f);
void* data = malloc(fsize);
fread(data, fsize, 1, f);
fclose(f);
TF_Buffer* buf = TF_NewBuffer();
buf->data = data;
buf->length = fsize;
buf->data_deallocator = free_buffer;
return buf;
}

The previous answer is your main option if you are wanting to use it outside of the TensorFlow project (and consequently not build with Bazel). You need to load it from the c_api.h with TF_GraphImportDef, I recommend training and doing testing in Python and then exporting the model/graph for use with C++/C Api when you have finished.

Related

Unable to compile tensorflow lite examples on adafruit circuitplayground bluefruit due to missing files in Adafruit_Tensorflow_Lite library

I am unable to compile the examples , hello_world_arcada and micro_speech_arcada shown below , on the adafruit website found here on my Circuit playground bluefruit microcontroller:
I installed the Adafruit_Tensorflow_Lite library as mentioned in the site however it turns out that examples cannot compile because they have numerous missing files. So i downloaded this tensorflow git hub repo and then transfered the missing files into the Adafruit_Tensorflow_Lite library.
I am now facing this error for the missing files : am_bsp.h ,am_mcu_apollo.h , am_util.h , i cannot locate these files on the repo or on google.[Note: i have found the am_bsp.h file in this repo
but it still doesnt compile.
Can anyone assist me in locating where i can find these files or a way to compile the example code mentioned in the adafruit website ?
The error is shown in the pic below of the missing file am_bsp.h when using Arduino IDE to compile:
My code is shown below:
#include <TensorFlowLite.h>
#include "Adafruit_TFLite.h"
#include "Adafruit_Arcada.h"
#include "output_handler.h"
#include "sine_model_data.h"
// Create an area of memory to use for input, output, and intermediate arrays.
// Finding the minimum value for your model may require some trial and error.
const int kTensorAreaSize (2 * 1024);
// This constant represents the range of x values our model was trained on,
// which is from 0 to (2 * Pi). We approximate Pi to avoid requiring additional
// libraries.
const float kXrange = 2.f * 3.14159265359f;
// Will need tuning for your chipset
const int kInferencesPerCycle = 200;
int inference_count = 0;
Adafruit_Arcada arcada;
Adafruit_TFLite ada_tflite(kTensorAreaSize);
// The name of this function is important for Arduino compatibility.
void setup() {
Serial.begin(115200);
//while (!Serial) yield();
arcada.arcadaBegin();
// If we are using TinyUSB we will have the filesystem show up!
arcada.filesysBeginMSD();
arcada.filesysListFiles();
// Set the display to be on!
arcada.displayBegin();
arcada.setBacklight(255);
arcada.display->fillScreen(ARCADA_BLUE);
if (! ada_tflite.begin()) {
arcada.haltBox("Failed to initialize TFLite");
while (1) yield();
}
if (arcada.exists("model.tflite")) {
arcada.infoBox("Loading model.tflite from disk!");
if (! ada_tflite.loadModel(arcada.open("model.tflite"))) {
arcada.haltBox("Failed to load model file");
}
} else if (! ada_tflite.loadModel(g_sine_model_data)) {
arcada.haltBox("Failed to load default model");
}
Serial.println("\nOK");
// Keep track of how many inferences we have performed.
inference_count = 0;
}
// The name of this function is important for Arduino compatibility.
void loop() {
// Calculate an x value to feed into the model. We compare the current
// inference_count to the number of inferences per cycle to determine
// our position within the range of possible x values the model was
// trained on, and use this to calculate a value.
float position = static_cast<float>(inference_count) /
static_cast<float>(kInferencesPerCycle);
float x_val = position * kXrange;
// Place our calculated x value in the model's input tensor
ada_tflite.input->data.f[0] = x_val;
// Run inference, and report any error
TfLiteStatus invoke_status = ada_tflite.interpreter->Invoke();
if (invoke_status != kTfLiteOk) {
ada_tflite.error_reporter->Report("Invoke failed on x_val: %f\n",
static_cast<double>(x_val));
return;
}
// Read the predicted y value from the model's output tensor
float y_val = ada_tflite.output->data.f[0];
// Output the results. A custom HandleOutput function can be implemented
// for each supported hardware target.
HandleOutput(ada_tflite.error_reporter, x_val, y_val);
// Increment the inference_counter, and reset it if we have reached
// the total number per cycle
inference_count += 1;
if (inference_count >= kInferencesPerCycle) inference_count = 0;
}
Try to install the library from below link, it should solve your problems,
https://github.com/tensorflow/tflite-micro-arduino-examples#how-to-install

Webm (VP8 / Opus) file read and write back

I am trying to develop a webrtc simulator in C/C++. For media handling, I plan to use libav. I am thinking of below steps to realize media exchange between two webrtc simulator. Say I have two webrtc simulators A and B.
Read media at A from a input webm file using av_read_frame api.
I assume I will get the encoded media (audio / video) data, am I correct here?
Send the encoded media data to simulator B over a UDP socket.
Simulator B receives the media data in UDP socket as RTP packets.
Simulator B extracts audio/video data from just received RTP packet.
I assume the extracted media data at simulator B are the encoded data only (am I correct here). I do not want to decode it. I want to write it to a file. Later I will play the file to check if I have done everything right.
To simplify this problem lets take out UDP socket part. Then my question reduces to read data from a webm input file, get the encoded media, prepare the packet and write to a output file using av_interleaved_write_frame or any other appropriate api. All these things I want to do using libav.
Is there any example code I can refer.
Or can somebody please guide me to develop it.
I am trying with a test program. As a first step, my aim is to read from a file and write to an output file. I have below code, but it is not working properly.
//#define _AUDIO_WRITE_ENABLED_
#include "libavutil/imgutils.h"
#include "libavutil/samplefmt.h"
#include "libavformat/avformat.h"
static AVPacket pkt;
static AVFormatContext *fmt_ctx = NULL;
static AVFormatContext *av_format_context = NULL;
static AVOutputFormat *av_output_format = NULL;
static AVCodec *video_codec = NULL;
static AVStream *video_stream = NULL;
static AVCodec *audio_codec = NULL;
static AVStream *audio_stream = NULL;
static const char *src_filename = NULL;
static const char *dst_filename = NULL;
int main (int argc, char **argv)
{
int ret = 0;
int index = 0;
if (argc != 3)
{
printf("Usage: ./webm input_video_file output_video_file \n");
exit(0);
}
src_filename = argv[1];
dst_filename = argv[2];
printf("Source file = %s , Destination file = %s\n", src_filename, dst_filename);
av_register_all();
/* open input file, and allocate format context */
if (avformat_open_input(&fmt_ctx, src_filename, NULL, NULL) < 0)
{
fprintf(stderr, "Could not open source file %s\n", src_filename);
exit(1);
}
/* retrieve stream information */
if (avformat_find_stream_info(fmt_ctx, NULL) < 0)
{
fprintf(stderr, "Could not find stream information\n");
exit(2);
}
av_output_format = av_guess_format(NULL, dst_filename, NULL);
if(!av_output_format)
{
fprintf(stderr, "Could not guess output file format\n");
exit(3);
}
av_output_format->audio_codec = AV_CODEC_ID_VORBIS;
av_output_format->video_codec = AV_CODEC_ID_VP8;
av_format_context = avformat_alloc_context();
if(!av_format_context)
{
fprintf(stderr, "Could not allocation av format context\n");
exit(4);
}
av_format_context->oformat = av_output_format;
strcpy(av_format_context->filename, dst_filename);
video_codec = avcodec_find_encoder(av_output_format->video_codec);
if (!video_codec)
{
fprintf(stderr, "Codec not found\n");
exit(5);
}
video_stream = avformat_new_stream(av_format_context, video_codec);
if (!video_stream)
{
fprintf(stderr, "Could not alloc stream\n");
exit(6);
}
avcodec_get_context_defaults3(video_stream->codec, video_codec);
video_stream->codec->codec_id = AV_CODEC_ID_VP8;
video_stream->codec->codec_type = AVMEDIA_TYPE_VIDEO;
video_stream->time_base = (AVRational) {1, 30};
video_stream->codec->width = 640;
video_stream->codec->height = 480;
video_stream->codec->pix_fmt = PIX_FMT_YUV420P;
video_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER;
video_stream->codec->bit_rate = 400000;
video_stream->codec->gop_size = 10;
video_stream->codec->max_b_frames=1;
#ifdef _AUDIO_WRITE_ENABLED_
audio_codec = avcodec_find_encoder(av_output_format->audio_codec);
if (!audio_codec)
{
fprintf(stderr, "Codec not found audio codec\n");
exit(5);
}
audio_stream = avformat_new_stream(av_format_context, audio_codec);
if (!audio_stream)
{
fprintf(stderr, "Could not alloc stream for audio\n");
exit(6);
}
avcodec_get_context_defaults3(audio_stream->codec, audio_codec);
audio_stream->codec->codec_id = AV_CODEC_ID_VORBIS;
audio_stream->codec->codec_type = AVMEDIA_TYPE_AUDIO;
audio_stream->time_base = (AVRational) {1, 30};
audio_stream->codec->sample_rate = 8000;
audio_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER;
#endif
if(!(av_output_format->flags & AVFMT_NOFILE))
{
if (avio_open(&av_format_context->pb, dst_filename, AVIO_FLAG_WRITE) < 0)
{
fprintf(stderr, "Could not open '%s'\n", dst_filename);
}
}
/* Before avformat_write_header set the stream */
avformat_write_header(av_format_context, NULL);
/* initialize packet, set data to NULL, let the demuxer fill it */
av_init_packet(&pkt);
pkt.data = NULL;
pkt.size = 0;
pkt.stream_index = video_stream->index;
ret = av_read_frame(fmt_ctx, &pkt);
while (ret >= 0)
{
index++;
//pkt.stream_index = video_avstream->index;
if(pkt.stream_index == video_stream->index)
{
printf("Video: Read cycle %d, bytes read = %d, pkt stream index=%d\n", index, pkt.size, pkt.stream_index);
av_write_frame(av_format_context, &pkt);
}
#ifdef _AUDIO_WRITE_ENABLED_
else if(pkt.stream_index == audio_stream->index)
{
printf("Audio: Read cycle %d, bytes read = %d, pkt stream index=%d\n", index, pkt.size, pkt.stream_index);
av_write_frame(av_format_context, &pkt);
}
#endif
av_free_packet(&pkt);
ret = av_read_frame(fmt_ctx, &pkt);
}
av_write_trailer(av_format_context);
/** Exit procedure starts */
avformat_close_input(&fmt_ctx);
avformat_free_context(av_format_context);
return 0;
}
When I execute this program, it outputs "codec not found". Now sure whats going wrong, Can somebody help please.
Codec not found issue is resolved by separately building libvpx1.4 version. Still struggling to read from source file, and writing to a destination file.
EDIT 1: After code modification, only video stuff I am able to write to a file, though some more errors are still present.
EDIT 2: With modified code (2nd round), I see video frames are written properly. For audio frames I added the code under a macro _AUDIO_WRITE_ENABLED_ , but if I enable this macro program crashing. Can somebody guide whats wrong in audio write part (code under macro _AUDIO_WRITE_ENABLED_).
I am not fully answering your question, but I hope we will get to the final solution eventually. When I tried to run your code, I got this error "time base not set".
Time base and other header specs are part of codec. This is, how I have this thing specified for writing into file (vStream is of AVStream):
#if LIBAVCODEC_VER_AT_LEAST(53, 21)
avcodec_get_context_defaults3(rc->vStream->codec, AVMEDIA_TYPE_VIDEO);
#else
avcodec_get_context_defaults2(rc->vStream->codec, AVMEDIA_TYPE_VIDEO);
#endif
#if LIBAVCODEC_VER_AT_LEAST(54, 25)
vStream->codec->codec_id = AV_CODEC_ID_VP8;
#else
vStream->codec->codec_id = CODEC_ID_VP8;
#endif
vStream->codec->codec_type = AVMEDIA_TYPE_VIDEO;
vStream->codec->time_base = (AVRational) {1, 30};
vStream->codec->width = 640;
vStream->codec->height = 480;
vStream->codec->pix_fmt = PIX_FMT_YUV420P;
EDIT: I ran your program in Valgrind and it segfaults on av_write_frame. Looks like its time_base and other specs for output are not set properly.
Add the specs before avformat_write_header(), before it is too late.

How to get array of float audio data from AudioQueueRef in iOS?

I'm working on getting audio into the iPhone in a form where I can pass it to a (C++) analysis algorithm. There are, of course, many options: the AudioQueue tutorial at trailsinthesand gets things started.
The audio callback, though, gives an AudioQueueRef, and I'm finding Apple's documentation thin on this side of things. Built-in methods to write to a file, but nothing where you actually peer inside the packets to see the data.
I need data. I don't want to write anything to a file, which is what all the tutorials — and even Apple's convenience I/O objects — seem to be aiming at. Apple's AVAudioRecorder (infuriatingly) will give you levels and write the data, but not actually give you access to it. Unless I'm missing something...
How to do this? In the code below there is inBuffer->mAudioData which is tantalizingly close but I can find no information about what format this 'data' is in or how to access it.
AudioQueue Callback:
void AudioInputCallback(void *inUserData,
AudioQueueRef inAQ,
AudioQueueBufferRef inBuffer,
const AudioTimeStamp *inStartTime,
UInt32 inNumberPacketDescriptions,
const AudioStreamPacketDescription *inPacketDescs)
{
static int count = 0;
RecordState* recordState = (RecordState*)inUserData;
AudioQueueEnqueueBuffer(recordState->queue, inBuffer, 0, NULL);
++count;
printf("Got buffer %d\n", count);
}
And the code to write the audio to a file:
OSStatus status = AudioFileWritePackets(recordState->audioFile,
false,
inBuffer->mAudioDataByteSize,
inPacketDescs,
recordState->currentPacket,
&inNumberPacketDescriptions,
inBuffer->mAudioData); // THIS! This is what I want to look inside of.
if(status == 0)
{
recordState->currentPacket += inNumberPacketDescriptions;
}
// so you don't have to hunt them all down when you decide to switch to float:
#define AUDIO_DATA_TYPE_FORMAT SInt16
// the actual sample-grabbing code:
int sampleCount = inBuffer->mAudioDataBytesCapacity / sizeof(AUDIO_DATA_TYPE_FORMAT);
AUDIO_DATA_TYPE_FORMAT *samples = (AUDIO_DATA_TYPE_FORMAT*)inBuffer->mAudioData;
Then you have your (in this case SInt16) array samples which you can access from samples[0] to samples[sampleCount-1].
The above solution did not work for me, I was getting the wrong sample data itself.(an endian issue) If incase someone is getting wrong sample data in future, I hope this helps you :
-(void)feedSamplesToEngine:(UInt32)audioDataBytesCapacity audioData:(void *)audioData {
int sampleCount = audioDataBytesCapacity / sizeof(SAMPLE_TYPE);
SAMPLE_TYPE *samples = (SAMPLE_TYPE*)audioData;
//SAMPLE_TYPE *sample_le = (SAMPLE_TYPE *)malloc(sizeof(SAMPLE_TYPE)*sampleCount );//for swapping endians
std::string shorts;
double power = pow(2,10);
for(int i = 0; i < sampleCount; i++)
{
SAMPLE_TYPE sample_le = (0xff00 & (samples[i] << 8)) | (0x00ff & (samples[i] >> 8)) ; //Endianess issue
char dataInterim[30];
sprintf(dataInterim,"%f ", sample_le/power); // normalize it.
shorts.append(dataInterim);
}

GNU Radio File Format for the recorded samples

Do you know the format in which GNU Radio ( File Sink in GNU Radio Companion) stores the samples in the Binary File?
I need to read these samples in Matlab, but the problem is the file is too big to be read in Matlab.
I am writing the program in C++ to read this binary file.
The file sink is just a dump of the data stream. If the data stream content was simple bytes then the content of the file is straightforward. If the data stream contained complex numbers then the file will contain a list of complex numbers where each complex number is given by two floats and each float by (usually) 4 bytes.
See the files gnuradio/gnuradio-core/src/lib/io/gr_file_sink.cc and gr_file_source.cc for the implementations of the gnuradio file reading and writing blocks.
You could also use python and gnuradio to convert the files into some other format.
from gnuradio import gr
# Assuming the data stream was complex numbers.
src = gr.file_source(gr.sizeof_gr_complex, "the_file_name")
snk = gr.vector_sink_c()
tb = gr.top_block()
tb.connect(src, snk)
tb.run()
# The complex numbers are then accessible as a python list.
data = snk.data()
Ben's answer still stands – but it's from a time long past (the module organization points at GNU Radio 3.6, I think). Organizationally, things are different now; data-wise, the File Sink remained the same.
GNU Radio now has relatively much block documentation in their wiki. In particular, the File Sink documentation page has a section on Handling File Sink data; not to overquote that:
// This is C++17
#include <algorithm>
#include <cmath>
#include <complex>
#include <cstddef>
#include <filesystem>
#include <fstream>
#include <string_view>
#include <vector>
#include <fmt/format.h>
#include <fmt/ranges.h>
using sample_t = std::complex<float>;
using power_t = float;
constexpr std::size_t read_block_size = 1 << 16;
int main(int argc, char *argv[]) {
// expect exactly one argument, a file name
if (argc != 2) {
fmt::print(stderr, "Usage: {} FILE_NAME", argv[0]);
return -1;
}
// just for convenience; we could as well just use `argv[1]` throughout the
// code
std::string_view filename(argv[1]);
// check whether file exists
if (!std::filesystem::exists(filename.data())) {
fmt::print(stderr, "file '{:s}' not found\n", filename);
return -2;
}
// calculate how many samples to read
auto file_size = std::filesystem::file_size(std::filesystem::path(filename));
auto samples_to_read = file_size / sizeof(sample_t);
// construct and reserve container for resulting powers
std::vector<power_t> powers;
powers.reserve(samples_to_read);
std::ifstream input_file(filename.data(), std::ios_base::binary);
if (!input_file) {
fmt::print(stderr, "error opening '{:s}'\n", filename);
return -3;
}
// construct and reserve container for read samples
// if read_block_size == 0, then read the whole file at once
std::vector<sample_t> samples;
if (read_block_size)
samples.resize(read_block_size);
else
samples.resize(samples_to_read);
fmt::print(stderr, "Reading {:d} samples…\n", samples_to_read);
while (samples_to_read) {
auto read_now = std::min(samples_to_read, samples.size());
input_file.read(reinterpret_cast<char *>(samples.data()),
read_now * sizeof(sample_t));
for (size_t idx = 0; idx < read_now; ++idx) {
auto magnitude = std::abs(samples[idx]);
powers.push_back(magnitude * magnitude);
}
samples_to_read -= read_now;
}
// we're not actually doing anything with the data. Let's print it!
fmt::print("Power\n{}\n", fmt::join(powers, "\n"));
}

Trying to parse OpenCV YAML ouput with yaml-cpp

I've got a series of OpenCv generated YAML files and would like to parse them with yaml-cpp
I'm doing okay on simple stuff, but the matrix representation is proving difficult.
# Center of table
tableCenter: !!opencv-matrix
rows: 1
cols: 2
dt: f
data: [ 240, 240]
This should map into the vector
240
240
with type float. My code looks like:
#include "yaml.h"
#include <fstream>
#include <string>
struct Matrix {
int x;
};
void operator >> (const YAML::Node& node, Matrix& matrix) {
unsigned rows;
node["rows"] >> rows;
}
int main()
{
std::ifstream fin("monsters.yaml");
YAML::Parser parser(fin);
YAML::Node doc;
Matrix m;
doc["tableCenter"] >> m;
return 0;
}
But I get
terminate called after throwing an instance of 'YAML::BadDereference'
what(): yaml-cpp: error at line 0, column 0: bad dereference
Abort trap
I searched around for some documentation for yaml-cpp, but there doesn't seem to be any, aside from a short introductory example on parsing and emitting. Unfortunately, neither of these two help in this particular circumstance.
As I understand, the !! indicate that this is a user-defined type, but I don't see with yaml-cpp how to parse that.
You have to tell yaml-cpp how to parse this type. Since C++ isn't dynamically typed, it can't detect what data type you want and create it from scratch - you have to tell it directly. Tagging a node is really only for yourself, not for the parser (it'll just faithfully store it for you).
I'm not really sure how an OpenCV matrix is stored, but if it's something like this:
class Matrix {
public:
Matrix(unsigned r, unsigned c, const std::vector<float>& d): rows(r), cols(c), data(d) { /* init */ }
Matrix(const Matrix&) { /* copy */ }
~Matrix() { /* delete */ }
Matrix& operator = (const Matrix&) { /* assign */ }
private:
unsigned rows, cols;
std::vector<float> data;
};
then you can write something like
void operator >> (const YAML::Node& node, Matrix& matrix) {
unsigned rows, cols;
std::vector<float> data;
node["rows"] >> rows;
node["cols"] >> cols;
node["data"] >> data;
matrix = Matrix(rows, cols, data);
}
Edit It appears that you're ok up until here; but you're missing the step where the parser loads the information into the YAML::Node. Instead, se it like:
std::ifstream fin("monsters.yaml");
YAML::Parser parser(fin);
YAML::Node doc;
parser.GetNextDocument(doc); // <-- this line was missing!
Matrix m;
doc["tableCenter"] >> m;
Note: I'm guessing dt: f means "data type is float". If that's the case, it'll really depend on how the Matrix class handles this. If you have a different class for each data type (or a templated class), you'll have to read that field first, and then choose which type to instantiate. (If you know it'll always be float, that'll make your life easier, of course.)