Calling fgets() on popen() of 'ssh' is flushing the beginning of stdin of the calling process (ptty issue) - ssh

I have now whittled this down to a minimal test case. Thus far I have been able to determine that this is an issue related to pseudo-terminals which come about with the pipe of ssh. Adding the '-t -t' to the ssh call improved things, in that now, it takes a second call to fgets() to cause the issue. I suspect that the stderr output of the ssh command somehow works into the issue, for now I have redirected stderr to stdout in the ssh code to execute. I do wonder if the "tcgetattr: Invalid argument" error is part of the problem, but am not sure how to get rid of that. It seems to come from the -t -t being present. I believe the -t -t is moving in the right direction, but I have to set up the pseudo terminal for stderr somehow and perhaps the test will work properly?
The Makefile:
test:
gcc -g -DBUILD_MACHINE='"$(shell hostname)"' -c -o test.o test.c
gcc -g -o test test.o
.PHONY: clean
clean:
rm -rf test.o test
The test.c source file:
#include <unistd.h>
#include <string.h>
#include <stdio.h>
int
main(int argc, char *argv[])
{
const unsigned int bufSize = 32;
char buf1[bufSize];
char buf2[bufSize];
int ssh = argv[1][0] == 'y';
const char *cmd = ssh ? "ssh -t -t " BUILD_MACHINE " \"ls\" 2>&1" : "ls";
FILE *fPtr = popen(cmd, "r");
if (fPtr == NULL) {
fprintf(stderr,"Unable to spawn command.\n");
perror("popen(3)");
exit(1);
}
printf("Command: %s\n", cmd);
if (feof(fPtr) == 0 && fgets(buf2, bufSize, fPtr) != NULL) {
printf("First result: %s\n", buf2);
if (feof(fPtr) == 0 && fgets(buf2, bufSize, fPtr) != NULL) {
printf("Second result: %s\n", buf2);
int nRead = read(fileno(stdin), buf1, bufSize);
if (nRead == 0) {
printf("???? popen() of ssh consumed the beginning of stdin ????\n");
} else if (nRead > 0) {
if (strncmp("The quick brown fox jumped", buf1, 26) != 0) {
printf("??? Failed ???\n");
} else {
printf("!!!!!!! Without ssh popen() did not consume stdin !!!!!!!\n");
}
}
}
}
}
This shows it running the passing way:
> echo "The quick brown fox jumped" | ./test n
Command: ls
First result: ARCH.linux_26_i86
Second result: Makefile
!!!!!!! Without ssh popen() did not consume stdin !!!!!!!
This shows it running the failing way:
> echo "The quick brown fox jumped" | ./test y
Command: ssh -t -t hostname "ls" 2>&1
First result: tcgetattr: Invalid argument
Second result: %backup%~ gmon.out
???? popen() of ssh consumed the beginning of stdin ????

Okay, I have got this working finally. The secret was to supply /dev/null as the input to my ssh command as follows from the test case above:
const char *cmd
= ssh ? "ssh -t -t " BUILD_MACHINE " \"ls\" 2>&1 < /dev/null" : "ls";
However, while the code works correctly, I get a nasty message which apparently I can ignore for my purposes (although I'd like to make the message go away):
tcgetattr: Inappropriate ioctl for device

Related

Prevent CUDA-enabled MPI from checking for CUDA devices

The OpenMPI 4.0.5 on our cluster is built with CUDA support, but I want to benchmark pnetcdf without needing CUDA for that. Since I want to do a number of test runs that I can start on like 1/4th of a node and my tests won't make use of the GPUs I wanted to ask if there is a way to suppress the MPI check for CUDA devices. Because when I simply obtain a SLURM allocation without GPUs, I get lots of errors from that alone.
These errors come from hwloc, and can be suppressed with HWLOC_HIDE_ERRORS=1 but I'd like to know if there is a more specific method.
Steps to reproduce:
frontend$ salloc -n 16 -t 8:00:00 -A k20200
node$ exec bash -l
node$ module load gcc openmpi
node$ mpicc -o /tmp/hello ~/usr/src/helloworld_mpi.c
node$ srun -n 1 /tmp/hello
CUDA: Failed to get number of devices with cudaGetDeviceCount(): no CUDA-capable device is detected
Hello world!, I'm rank 0 of 1!
node$ HWLOC_HIDE_ERRORS=1 srun -n 1 /tmp/hello
Hello world!, I'm rank 0 of 1!
node$ logout
The example code used above is the following but any program without CUDA use is equally useful in this exercise
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#define xmpi(rc) \
do { \
int err = (rc); \
if (err != MPI_SUCCESS) { \
char msg[MPI_MAX_ERROR_STRING + 1]; \
int msg_len; \
\
if (MPI_Error_string(err, msg, &msg_len) \
== MPI_SUCCESS){ \
msg[msg_len] = '\0'; \
\
fprintf(stderr, \
"Problem in MPI call: %d = %s\n", \
err, msg); \
MPI_Abort(MPI_COMM_WORLD, 1); \
} \
} \
} while (0)
int
main(int argc, char *argv[])
{
xmpi(MPI_Init(&argc, &argv));
int rank, size;
xmpi(MPI_Comm_rank(MPI_COMM_WORLD, &rank));
xmpi(MPI_Comm_size(MPI_COMM_WORLD, &size));
printf("Hello world!, I'm rank %d of %d!\n", rank, size);
xmpi(MPI_Finalize());
return EXIT_SUCCESS;
}

Opening a binary file handled by C++ std::ifstream in gem5 syscall emulation fails is_open check

I have mnist.loader.hpp header file that opens and reads t10k-images-idx3-ubyte and t10k-labels-idx1-ubyte binary files in a given path using std::ifstream as can be seen in the snippet below:
std::string imagePath = dataDir + std::string("t10k-images-idx3-ubyte");
std::string labelPath = dataDir + std::string("t10k-labels-idx1-ubyte");
std::ifstream imageStream(imagePath, std::ios::binary);
std::ifstream labelStream(labelPath, std::ios::binary);
The path of the directory having those two binary files dataDir is embedded in mnist_caffe.cpp. It could be changed as per my setting in the std::string dataDir = "data/" line.
I set dataDir to /home/mohamed/, then compiled and statically-linked an aarch64 executable out of them and my target is to simulate it on gem5.
The problem:
gem5 always fails to read the files. It gives "Failed to read /home/mohamed/t10k-images-idx3-ubyte".
If we have a look once again at mnist_loader.hpp, we will notice that this error message is due to either the file not being open as or being opened but can't be read :
if (!imageStream.is_open())
{
std::cerr << "Failed to load " << imagePath << std::endl;
return nullptr;
}
or
imageStream.read(reinterpret_cast<char*>(&magic), sizeof(magic));
if (magic != 0x03080000)
{
std::cerr << "Failed to read " << imagePath << std::endl;
return nullptr;
}
or
if (!imageStream.good())
{
std::cerr << "Failed to read " << imagePath << std::endl;
return nullptr;
}
My guess is that gem5 can not open the file in the first place!
My trials:
(I am only interested in opening and reading t10k-images-idx3-ubyte for now)
1- Trying to pass the file to the stdin using -i option: $ ./build/ARM/gem5.opt ./configs/example/se.py -c mnist_caffe -i '/home/mohamed/t10k-images-idx3-ubyte'
2- Using -o: $ ./build/ARM/gem5.opt ./configs/example/se.py -c mnist_caffe -o "-i /home/mohamed/t10k-images-idx3-ubyte"
3- Using --redirects: $ ./build/ARM/gem5.opt ./configs/example/se.py -c /home/mohamed/NNDeploy/ML-examples/armnn-mnist/mnist_caffe --redirects /home/mohamed/t10k-images-idx3-ubyte=/home/mohamed/t10k-images-idx3-ubyte
4- Opening the binary file in the host system using xxd and then passing it to stdin: $ ./build/ARM/gem5.opt ./configs/example/se.py -c ~/NNDeploy/ML-examples/armnn-mnist/mnist_caffe -o " -i /usr/bin/xxd /home/mohamed/t10k-images-idx3-ubyte"
All the above resulted in the same failed-to-read message. I ran out of ideas and it would be great if someone could provide some suggestions!

How to code "git commit" in libgit2?

I have searched Google and Stackoverflow for an answer to the question of how to code the equivalent of
git commit -a -m "message"
in libgit2 (https://libgit2.github.com) and C or C++.
But I cannot find a ready and working answer to this question.
I am using libgit2-0.21.
Below is code that initializes a git repository, adds two files to it, and stages the two files so they are ready to be committed.
My question is how to code "git commit -a -m "msg" in libgit2?
#include <sys/stat.h>
#include <string>
#include <fstream>
#include <iostream>
#include <git2.h>
using namespace std;
int main (int argc, char** argv)
{
git_threads_init ();
// Create repository directory.
string directory = "repository";
mkdir (directory.c_str(), 0777);
// Initialize the repository: git init.
git_repository *repo = NULL;
int result = git_repository_init (&repo, directory.c_str(), false);
if (result != 0) cerr << giterr_last ()->message << endl;
// Store two files in the repository directory.
ofstream file;
file.open ("repository/file1", ios::binary | ios::trunc);
file << "Contents of file one";
file.close ();
file.open ("repository/file2", ios::binary | ios::trunc);
file << "Contents of file two";
file.close ();
// Run the equivalent of "git add ."
// Get the git index.
git_index * index = NULL;
result = git_repository_index (&index, repo);
if (result != 0) cerr << giterr_last ()->message << endl;
// Add all files to the git index.
result = git_index_add_all (index, NULL, 0, NULL, NULL);
if (result != 0) cerr << giterr_last ()->message << endl;
// Write the index to disk.
result = git_index_write (index);
if (result != 0) cerr << giterr_last ()->message << endl;
// Run the equivalent of "git commit -a -m "commit message".
// How to do that through libgit2?
// Run "git status" to see the result.
system ("cd repository; git status");
// Free resources.
git_index_free (index);
git_repository_free (repo);
git_threads_shutdown ();
return 0;
}
The code can be compiled as follows:
g++ -Wall -I/opt/local/include -L/opt/local/lib -lgit2 -o test git.cpp
Below is the output of running the compiled binary:
On branch master
Initial commit
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: file1
new file: file2
Once the index is updated
create a tree from it through git_index_write_tree()
create a commit that references this tree through git_commit_create_v()
See this end to end test which performs the equivalent of the following
$ echo "test" > test.txt
$ git add .
$ git commit -m "Initial commit"

awk command to run a C++ code and input several times

Say, I have a C code which I compile like:
$ gcc code.c -o f.out
$ ./f.out inputfile outputfile
Then the code asks for input
$ enter mass:
Now if I need to run this code for example 200 times and the input files have name : 0c.txt, 1c.txt, ....., 199c.txt etc and I want to use same value of mass every time (e.g. mass=6) then how do I write an "awk" command for that? Thanks for your help.
You don't specify your outputfile name. I'll assume 0c.out, 1c.out, ...
I'm also assuming that the f.out program reads the mass from stdin instead of anything more complicated.
#!/usr/bin/gawk -f
BEGIN {
mass = 6
for (i=0; i<200; i++) {
cmd = sprintf("./f.out %dc.txt %dc.out", i, i)
print mass |& cmd
close(cmd, "to")
while ((cmd |& getline out) > 0) {
do something with each line of output from ./f.out
}
close(cmd)
}
}
ref http://www.gnu.org/software/gawk/manual/html_node/Two_002dway-I_002fO.html
In bash, you'd write:
for i in $(seq 0 199); do
echo 6 | ./f.out ${i}c.txt ${i}c.out
done

Error during nfq_open()

I'm trying to set up an nfq listener based on This bit of code:
main (int argc, char **argv)
{
struct nfq_handle *h;
struct nfq_q_handle *qh;
struct nfnl_handle *nh;
int fd;
int rv;
char buf[4096] __attribute__ ((aligned));
printf ("opening library handle\n");
h = nfq_open ();
if (!h)
{
fprintf (stderr, "error during nfq_open()\n");
exit (1);
}
...
I have the following iptables rule set:
NFQUEUE udp -- 0.0.0.0/0 192.168.50.0/24 udp dpt:10000 NFQUEUE num 5061
But running the program always gives: Error during nfq_open()
Is there something incorrect with my IPtables rule? I used the following command:
/sbin/iptables -A FORWARD --protocol udp --dport 10000 -j NFQUEUE --queue-num 5061 -d 192.168.50.0/24
Thanks.
Please add "-t" switch with the table name "nat|filter|mangle" in which you want to add rule.
This won't be an issue with the iptables command at all. I would guess that you are not running the program as root, which is required to use NFQUEUE.