Code Sample for Reading Values from Hyper-V KVP component on Linux (aka KVP Data Exchange) - hyper-v

Hyper-V includes a KVP component that transmits key / value pairs between the host and a guest VM.
Code samples for sending and receiving values are available for Windows Guests in PowerShell in WMI.
However, my guest is using a Linux version of this service.
Where can I find a sample Linux script that queries this service for key / value pairs?

Key details from a blog entry I wrote covering the problem. (I could not find the answer elsewhere):
First, make sure you have the KVP service installed.
KVP data is transferred to the Linux file system through the collaboration of a kernel driver and a user mode daemon.
The [KVP driver code], hv_kvp.c, is compiled into the hv_util kernel module Source). Since the driver is part of the Linux kernel code, it is provided by default with recent versions of common Linux distributions. E.g.
[root#centos6-4-hv ~]# cat /etc/*-release
CentOS release 6.4 (Final)
CentOS release 6.4 (Final)
CentOS release 6.4 (Final)
[root#centos6-4-hv ~]# modinfo -F filename hv_utils
/lib/modules/2.6.32-358.el6.i686/kernel/drivers/hv/hv_utils.ko
However, it is the usermode daemon, hv_kvp_daemon, that copies KVP data to the system. On startup, hv_kvp_daemon creates files to store kvp data under
/var/lib/hyperv (source). Each file is known as a 'pool', and there is a file for each data pool. E.g.
[root#centos6-4-hv hyperv]# ls -al /var/lib/hyperv/
total 36
drwxr-xr-x. 2 root root 4096 Sep 11 21:33 .
drwxr-xr-x. 16 root root 4096 Sep 10 13:59 ..
-rw-r--r--. 1 root root 2560 Sep 10 17:05 .kvp_pool_0
-rw-r--r--. 1 root root 0 Sep 10 14:02 .kvp_pool_1
-rw-r--r--. 1 root root 0 Sep 10 14:02 .kvp_pool_2
-rw-r--r--. 1 root root 28160 Sep 10 14:02 .kvp_pool_3
-rw-r--r--. 1 root root 0 Sep 10 14:02 .kvp_pool_4
The prefix of each file is the pool number. This corresponds to the KVP source. E.g. remember that source '0' is used for transmitting data from host to guest? That means our KVP data is in /var/lib/hyperv/.kvp_pool_0. E.g.
[root#centos6-4-hv hyperv]# cat /var/lib/hyperv/.kvp_pool_0
cloudstack-vm-userdatausername=root;password=1pass#word1[root#centos6-4-hv hyperv]#
These KVP data files contain an array of key / value pairs. Each is a byte array of a fixed size. (source)
/*
* Maximum key size - the registry limit for the length of an entry name
* is 256 characters, including the null terminator
*/
#define HV_KVP_EXCHANGE_MAX_KEY_SIZE (512)
/*
* bytes, including any null terminators
*/
#define HV_KVP_EXCHANGE_MAX_VALUE_SIZE (2048)
The byte array contains a UTF-8 encoded string, which is padded out to the max size with null characters. However, null string termination is not guaranteed (see kvp_send_key).
Provided there is only one key and the key name known, the easiest way to parse the file is to use sed. To remove null characters and the key name used in our example, you would use the following:
[root#centos6-4-hv hyperv]# cat /var/lib/hyperv/.kvp_pool_0 | sed 's/\x0//g' | sed 's/cloudstack-vm-userdata//g' > userdata
[root#centos6-4-hv hyperv]# more userdata
username=root;password=1pass#word1

Here is a bash script to read the key-value pairs for a given file:
#!/bin/bash
fname=$1
echo "Reading $fname"
nb=$(wc -c < $1)
nkv=$(( nb / (512+2048) ))
for n in $(seq 0 $(( $nkv - 1 )) ); do
offset=$(( $n * (512 + 2048) ))
k=$(dd if=$fname count=512 bs=1 skip=$offset status=none | sed 's/\x0.*//g')
v=$(dd if=$fname count=2048 bs=1 skip=$(( $offset + 512 )) status=none | sed 's/\x0.*//g')
echo "$k = $v"
done

Same functionality in java:
String guest_param_file="/var/lib/hyperv/.kvp_pool_3";
if (new File(guest_param_file).exists())
{
try
{
BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(guest_param_file),"UTF-8"));
char [] ckey=new char[512] ;
char [] cvalue=new char[2048] ;
while (true)
{
int charcount=br.read(ckey);
if (charcount==-1)
{
break;
}
br.read(cvalue);
String key=new String(ckey).trim();
String value=new String(cvalue).trim();
System.out.println( key+" = "+value);
}
br.close();
}
catch (UnsupportedEncodingException ex)
{
}
catch (FileNotFoundException ex)
{
}
catch (IOException ex)
{
}
}

Related

Why doesn't 'utf8-c8' encoding work when reading filehandles

I wish to read byte sequences that will not decode as valid UTF-8, specifically byte sequences that correspond to high and low surrogates code points. The result should be a raku string.
I read that, in raku, the 'utf8-c8' encoding can be used for this purpose.
Consider code point U+D83F. It is a high surrogate (reserved for the high half of UTF-16 surrogate pairs).
U+D83F has a byte sequence of 0xED 0xA0 0xBF, if encoded as UTF-8.
Slurping a file? Works
If I slurp a file containing this byte sequence, using 'utf8-c8' as the encoding, I get the expected result:
echo -n $'\ud83f' >testfile # Create a test file containing the byte sequence
myprog1.raku:
#!/usr/local/bin/raku
$*OUT.encoding('utf8-c8');
print slurp('testfile', enc => 'utf8-c8');
$ ./myprog1.raku | od -An -tx1
ed a0 bf
✔️ expected result
Slurping a filehandle? Doesn't work
But if I switch from slurping a file path to slurping a filehandle, it doesn't work, even though I set the filehandle's encoding to 'utf8-c8':
myprog2.raku
#!/usr/local/bin/raku
$*OUT.encoding('utf8-c8');
my $fh = open "testfile", :r, :enc('utf8-c8');
print slurp($fh, enc => 'utf8-c8');
#print $fh.slurp; # I tried this too: same error
$ ./myprog2.raku
Error encoding UTF-8 string: could not encode Unicode Surrogate codepoint 55359 (0xD83F)
in block <unit> at ./myprog2.raku line 4
Environment
Edit 2022-10-30: I originally used my distro's package (Fedora Linux 36: Rakudo version 2020.07). I just downloaded the latest Rakudo binary release (2022.07-01). Result was the same.
$ /usr/local/bin/raku --version
Welcome to Rakudo™ v2022.07.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2022.07.
$ uname -a
Linux hx90 5.19.16-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Oct 16 22:50:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: Fedora
Description: Fedora release 36 (Thirty Six)
Release: 36
Codename: ThirtySix

Golang weird behavior using filepath.Glob

I'm confused about glob usage on Golang, I'm probably missing some environment variable. I don't know if I'm doing it right.
This code works perfectly fine when ran on my IDE (Intellij IDEA), but when this is ran on the OS via go run it doesn't work. I can't figure out what is the difference.
package main
import (
"path/filepath"
"fmt"
"os"
)
func main() {
file := os.Args[1]
matches, err := filepath.Glob(file)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println(fmt.Sprintf("Number of matches:%d", len(matches)))
fmt.Println(matches)
}
Running on OS
go run globtest.go /Users/bernardovale/glotest/*.bkp
Number of matches:1
[/Users/bernardovale/glotest/test1.bkp]
ls -l /Users/bernardovale/glotest/*.bkp
-rw-r--r-- 1 bernardovale staff 0 May 27 12:06 /Users/bernardovale/glotest/test1.bkp
-rw-r--r-- 1 bernardovale staff 0 May 27 12:06 /Users/bernardovale/glotest/test2.bkp
-rw-r--r-- 1 bernardovale staff 0 May 27 12:06 /Users/bernardovale/glotest/test3.bkp
Running on IntelliJ IDEA
The difference here is that the shell is executing the glob and providing the individual values to your application. When executing from a shell you should wrap a glob in double quotes to make sure it's not evaluated by your shell first. See the example below.
Seans-MBP-2:~ sthorne$ echo Testing*
Testing Testing2 Testing3
Seans-MBP-2:~ sthorne$ echo "Testing*"
Testing*

Redis server console output clarification?

I'm looking at the redis output console and I'm trying to understand the displayed info :
(didn't find that info in the quick guide)
So redis-server.exe outputs this :
/*1*/ [2476] 24 Apr 11:46:28 # Open data file dump.rdb: No such file or directory
/*2*/ [2476] 24 Apr 11:46:28 * The server is now ready to accept connections on port 6379
/*3*/ [2476] 24 Apr 11:42:35 - 1 clients connected (0 slaves), 1188312 bytes in use
/*4*/ [2476] 24 Apr 11:42:40 - DB 0: 1 keys (0 volatile) in 4 slots HT.
Regarding line #1 - what does the dump.rdb file is used for ? is it the data itself ?
what is the [2476] number ? it is not a port since line #2 tells port is 6379
What does (0 slaves) means ?
in line #3 - 1188312 bytes used - but what is the max value so i'd know overflows ...? is it for whole databases ?
Line #3 What does (0 volatile) means ?
Line #4 - why do i have 4 slots HT ? I have no data yet
[2476] - process ID
dump.rdb - redis can persist data by snapshoting, dump.rdb is the default file name http://redis.io/topics/persistence
0 slaves - redis can work in master-slave mode, 0 slaves informs you that there are no slave servers connected
1188312 bytes in use - total number of bytes allocated by Redis using its allocator
0 volatile - redis can set keys with expiration time, this is the count of them
4 slots HT - current hash table size, initial table size is 4, as you add more items hash table will grow

How to prevent ._ (dot underscore) files?

I am using Textmate 2 to edit a rails project on remote Linux server via sshfs.
When I save a file (e.g. README.rdoc) there is another file created (i.e. ._README.rdoc):
-rw-rw-r-- 1 4096 Feb 17 17:19 ._README.rdoc
-rw-rw-r-- 1 486 Feb 17 17:19 README.rdoc
The Textmate doc mentioned how to disable extended attributes:
defaults write com.macromates.textmate OakDocumentDisableFSMetaData 1
but ._ files are still created after the above defaults write.
Is there a way to disable creation of ._ files when using sshfs + Textmate 2?
To disable Extended Attributes in Textmate 2, use:
defaults write com.macromates.TextMate.preview volumeSettings '{ "/Users/ohho/Mount/" = { extendedAttributes = 0; }; }'
Where /Users/ohho/Mount/ is the parent folder of all my sshfs mounted folders.
As I have tried with following command THIS IS NOT WORKING :
NOT WORKING:-"defaults write com.macromates.textmate OakDocumentDisableFSMetaData 1"
I have taken reference from :
https://github.com/textmate/textmate/wiki/Hidden-Settings
and its working fine now.
TextMate use extended attributes to store caret position, etc.
On file systems which don’t support extended attributes
(most network file systems), OS X will create an auxiliary file with
a dot-underscore prefix (e.g. ._filename).
If you don’t want these files, you can disable the use of extended
attributes. This is presently controlled with the volumeSettings key.
Its values are (1) an associative array with path prefix; and (2) another associative array with settings for that path. (Presently, only extendedAttributes
is supported.)
So, if we wanted to disable extended attributes for files under /net/:
defaults write com.macromates.TextMate.preview volumeSettings '{ "/net/" = { extendedAttributes = 0; }; }'

PhantomJS: exported PDF to stdout

Is there a way to trigger the PDF export feature in PhantomJS without specifying an output file with the .pdf extension? We'd like to use stdout to output the PDF.
You can output directly to stdout without a need for a temporary file.
page.render('/dev/stdout', { format: 'pdf' });
See here for history on when this was added.
If you want to get HTML from stdin and output the PDF to stdout, see here
Sorry for the extremely long answer; I have a feeling that I'll need to refer to this method several dozen times in my life, so I'll write "one answer to rule them all". I'll first babble a little about files, file descriptors, (named) pipes, and output redirection, and then answer your question.
Consider this simple C99 program:
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char* argv[])
{
if (argc < 2) {
printf("Usage: %s file_name\n", argv[0]);
return 1;
}
FILE* file = fopen(argv[1], "w");
if (!file) {
printf("No such file: %s\n", argv[1]);
return 2;
}
fprintf(file, "some text...");
fclose(file);
return 0;
}
Very straightforward. It takes an argument (a file name) and prints some text into it. Couldn't be any simpler.
Compile it with clang write_to_file.c -o write_to_file.o or gcc write_to_file.c -o write_to_file.o.
Now, run ./write_to_file.o some_file (which prints into some_file). Then run cat some_file. The result, as expected, is some text...
Now let's get more fancy. Type (./write_to_file.o /dev/stdout) > some_file in the terminal. We're asking the program to write to its standard output (instead of a regular file), and then we're redirecting that stdout to some_file (using > some_file). We could've used any of the following to achieve this:
(./write_to_file.o /dev/stdout) > some_file, which means "use stdout"
(./write_to_file.o /dev/stderr) 2> some_file, which means "use stderr, and redirect it using 2>"
(./write_to_file.o /dev/fd/2) 2> some_file, which is the same as above; stderr is the third file descriptor assigned to Unix processes by default (after stdin and stdout)
(./write_to_file.o /dev/fd/5) 5> some_file, which means "use your sixth file descriptor, and redirect it to some_file"
In case it's not clear, we're using a Unix pipe instead of an actual file (everything is a file in Unix after all). We can do all sort of fancy things with this pipe: write it to a file, or write it to a named pipe and share it between different processes.
Now, let's create a named pipe:
mkfifo my_pipe
If you type ls -l now, you'll see:
total 32
prw-r--r-- 1 pooriaazimi staff 0 Jul 15 09:12 my_pipe
-rw-r--r-- 1 pooriaazimi staff 336 Jul 15 08:29 write_to_file.c
-rwxr-xr-x 1 pooriaazimi staff 8832 Jul 15 08:34 write_to_file.o
Note the p at the beginning of second line. It means that my_pipe is a (named) pipe.
Now, let's specify what we want to do with our pipe:
gzip -c < my_pipe > out.gz &
It means: gzip what I put inside my_pipe and write the results in out.gz. The & at the end asks the shell to run this command in the background. You'll get something like [1] 10449 and the control gets back to the terminal.
Then, simply redirect the output of our C program to this pipe:
(./write_to_file.o /dev/fd/5) 5> my_pipe
Or
./write_to_file.o my_pipe
You'll get
[1]+ Done gzip -c < my_pipe > out.gz
which means the gzip command has finished.
Now, do another ls -l:
total 40
prw-r--r-- 1 pooriaazimi staff 0 Jul 15 09:14 my_pipe
-rw-r--r-- 1 pooriaazimi staff 32 Jul 15 09:14 out.gz
-rw-r--r-- 1 pooriaazimi staff 336 Jul 15 08:29 write_to_file.c
-rwxr-xr-x 1 pooriaazimi staff 8832 Jul 15 08:34 write_to_file.o
We've successfully gziped our text!
Execute gzip -d out.gz to decompress this gziped file. It will be deleted and a new file (out) will be created. cat out gets us:
some text...
which is what we expected.
Don't forget to remove the pipe with rm my_pipe!
Now back to PhantomJS.
This is a simple PhantomJS script (render.coffee, written in CoffeeScript) that takes two arguments: a URL and a file name. It loads the URL, renders it and writes it to the given file name:
system = require 'system'
renderUrlToFile = (url, file, callback) ->
page = require('webpage').create()
page.viewportSize = { width: 1024, height : 800 }
page.settings.userAgent = 'Phantom.js bot'
page.open url, (status) ->
if status isnt 'success'
console.log "Unable to render '#{url}'"
else
page.render file
delete page
callback url, file
url = system.args[1]
file_name = system.args[2]
console.log "Will render to #{file_name}"
renderUrlToFile "http://#{url}", file_name, (url, file) ->
console.log "Rendered '#{url}' to '#{file}'"
phantom.exit()
Now type phantomjs render.coffee news.ycombinator.com hn.png in the terminal to render Hacker News front page into file hn.png. It works as expected. So does phantomjs render.coffee news.ycombinator.com hn.pdf.
Let's repeat what we did earlier with our C program:
(phantomjs render.coffee news.ycombinator.com /dev/fd/5) 5> hn.pdf
It doesn't work... :( Why? Because, as stated on PhantomJS's manual:
render(fileName)
Renders the web page to an image buffer and save it
as the specified file.
Currently the output format is automatically set based on the file
extension. Supported formats are PNG, JPEG, and PDF.
It fails, simply because neither /dev/fd/2 nor /dev/stdout end in .PNG, etc.
But no fear, named pipes can help you!
Create another named pipe, but this time use the extension .pdf:
mkfifo my_pipe.pdf
Now, tell it to simply cat its inout to hn.pdf:
cat < my_pipe.pdf > hn.pdf &
Then run:
phantomjs render.coffee news.ycombinator.com my_pipe.pdf
And behold the beautiful hn.pdf!
Obviously you want to do something more sophisticated that just cating the output, but I'm sure it's clear now what you should do :)
TL;DR:
Create a named pipe, using ".pdf" file extension (so it fools PhantomJS to think it's a PDF file):
mkfifo my_pipe.pdf
Do whatever you want to do with the contents of the file, like:
cat < my_pipe.pdf > hn.pdf
which simply cats it to hn.pdf
In PhantomJS, render to this file/pipe.
Later on, you should remove the pipe:
rm my_pipe.pdf
As pointed out by Niko you can use renderBase64() to render the web page to an image buffer and return the result as a base64-encoded string.But for now this will only work for PNG, JPEG and GIF.
To write something from a phantomjs script to stdout just use the filesystem API.
I use something like this for images :
var base64image = page.renderBase64('PNG');
var fs = require("fs");
fs.write("/dev/stdout", base64image, "w");
I don't know if the PDF format for renderBase64() will be in a future version of phanthomjs but as a workaround something along these lines may work for you:
page.render(output);
var fs = require("fs");
var pdf = fs.read(output);
fs.write("/dev/stdout", pdf, "w");
fs.remove(output);
Where output is the path to the pdf file.
I don't know if it would address your problem, but you may also check the new renderBase64() method added to PhantomJS 1.6: https://github.com/ariya/phantomjs/blob/master/src/webpage.cpp#L623
Unfortunately, the feature is not documented on the wiki yet :/