I'm in the process of writing an NFS V4 client and am debugging the results with Wireshark. I'm unable to read a file.
Through OPEN followed by GETATTR, I've opened the file and confirmed it's the desired file by the matching length (1001 bytes).
I then try to READ a single byte of the file with offset 0 and length 1. The resulting reply returns 0 bytes of data, even though the EOF is false. Repeated calls to the read command yield the same result.
Is there parameters in open or read that I'm missing to actually read the file?
Wireshark Output
Open Call
Operations (count: 5): SEQUENCE, PUTROOTFH, OPEN, GETFH, GETATTR
Opcode: SEQUENCE (53)
Opcode: PUTROOTFH (24)
Opcode: OPEN (18)
seqid: 0x00000000
share_access: OPEN4_SHARE_ACCESS_READ (1)
share_deny: OPEN4_SHARE_DENY_NONE (0)
clientid: 0x13f5c375a578cd7c
owner: <DATA>
Open Type: OPEN4_NOCREATE (0)
Claim Type: CLAIM_NULL (0)
Opcode: GETFH (10)
Opcode: GETATTR (9)
Open Reply
Operations (count: 5)
Opcode: SEQUENCE (53)
Opcode: PUTROOTFH (24)
Opcode: OPEN (18)
Status: NFS4_OK (0)
StateID
[StateID Hash: 0x91a9]
StateID seqid: 1
StateID Other: 13f5c375a578cd7c00000000
[StateID Other hash: 0x5b73]
change_info
Atomic: Yes
changeid (before): 110
changeid (after): 110
result flags: 0x00000004, locktype posix
.... .... .... .... .... .... .... ..0. = confirm: False
.... .... .... .... .... .... .... .1.. = locktype posix: True
.... .... .... .... .... .... .... 0... = preserve unlinked: False
.... .... .... .... .... .... ..0. .... = may notify lock: False
Delegation Type: OPEN_DELEGATE_NONE (0)
Opcode: GETFH (10)
Status: NFS4_OK (0)
Filehandle
length: 128
[hash (CRC-32): 0xc5dcd623]
FileHandle: 2b3e5cee938ee2b6afff448601a384b8ffc30bab28b5e2a4...
Opcode: GETATTR (9)
Status: NFS4_OK (0)
Attr mask: 0x00000010 (Size)
reqd_attr: Size (4)
size: 1001
Read Call
Operations (count: 3): SEQUENCE, PUTFH, READ
Opcode: SEQUENCE (53)
Opcode: PUTFH (22)
FileHandle
length: 128
[hash (CRC-32): 0xc5dcd623]
FileHandle: 2b3e5cee938ee2b6afff448601a384b8ffc30bab28b5e2a4...
Opcode: READ (25)
StateID
[StateID Hash: 0x91a9]
StateID seqid: 1
StateID Other: 13f5c375a578cd7c00000000
[StateID Other hash: 0x5b73]
offset: 0
count: 1
Read Reply
Operations (count: 3)
Opcode: SEQUENCE (53)
Opcode: PUTFH (22)
Status: NFS4_OK (0)
Opcode: READ (25)
Status: NFS4_OK (0)
eof: 0
Read length: 0
Data: <EMPTY>
For anybody who runs into a similar situation, I was able to fix the problem by changing the cachethis flag in the SEQUENCE operation of the READ call from true to false.
struct SEQUENCE4args {
sessionid4 sa_sessionid;
sequenceid4 sa_sequenceid;
slotid4 sa_slotid;
slotid4 sa_highest_slotid;
bool sa_cachethis; // ensure this is false
};
Some of the behavior of the cachethis flag is described in RFC 5661, but the information does not include why this behavior occurred.
One possibility is that Amazon's implementation of the NFS spec has a bug in the behavior with sa_cachethis.
Related
I am working on a react-native Android app using react-native-ble-plx for BTLE support, and Windows 10 using the .NET API Windows.Devices.Bluetooth.GenericAttributeProfile for the GATT server/peripheral.
When I add any "ServiceData" to the advertising payload in the Windows GATT service, the scanned device's 'serviceData' payload is always null in the react-native client.
In the Windows 10 server, if the GattServiceProviderAdvertisingParameters.ServiceData property is not null, then the service advertisement status is GattServiceProviderAdvertisementStatus.StartedWithoutAllAdvertisementData. (When leaving 'ServiceData' set to null, it is GattServiceProviderAdvertisementStatus.Started, as expected).
Using the Silicon Labs "EFR Connect" app in android, it also does not show any ServiceData for the device in the advertising packet.
Using WireShark with BTVS to inspect the packets on the Windows machine, it does show the Service Data bytes, and the controller shows a 'Success' response.
Here is the code where I set up the ServiceData in Windows:
...
GattServiceProviderAdvertisingParameters advParameters = new GattServiceProviderAdvertisingParameters
{
IsConnectable = _peripheralSupported,
IsDiscoverable = true,
ServiceData = GetServiceData().AsBuffer()
};
_serviceProvider.AdvertisementStatusChanged += ServiceProvider_AdvertisementStatusChanged;
_serviceProvider.StartAdvertising(advParameters);
...
private byte[] GetServiceData()
{
var flagsData = new List<byte> { 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
flagsData.AddRange(new List<byte>(Encoding.ASCII.GetBytes("ABCDEFGHIJKLMNOP")));
return flagsData.ToArray();
}
Here is the code in the react-native scanner:
public scanDevices(
timeoutSeconds: number,
listener: (error: string | null, scannedDevice: IPhoenixDevice | null) => void,
): void {
console.log('Entered PhoenixDeviceManager.scanDevices');
this.isScanning = true;
try {
this.scannedDevices.length = 0; // Clear array of devices
this.bleManager.startDeviceScan(this.serviceUUIDs, null, (error, scannedDevice) => {
console.log('In device scan callback');
if (error) {
console.warn(error);
listener(`Error message: ${error.message}, reason: ${error.reason}`, null);
}
if (!scannedDevice) {
console.log('scannedDevice is null');
} else if (!this.scannedDevices.some((d) => d.id === scannedDevice.id)) {
this.scannedDevices.push(scannedDevice);
listener(null, new PhoenixDevice(scannedDevice.id, scannedDevice.name));
console.log(
`Device discovered, id: ${scannedDevice.id}, name: ${scannedDevice.name}, localName: ${scannedDevice.localName}, serviceData: ${scannedDevice.serviceData}`,
);
}
});
const timeoutMs = timeoutSeconds * 1000;
// stop scanning devices after specified number of seconds
setTimeout(() => {
this.stopScanning();
}, timeoutMs);
} catch (error) {
console.error(error);
this.stopScanning();
}
}
Here is the WireShark trace running on Windows 10 showing the packet containing the ServiceData (and the Success response):
Frame 499: 84 bytes on wire (672 bits), 84 bytes captured (672 bits) on interface TCP#127.0.0.1:24352, id 0
Bluetooth
Bluetooth HCI H4
Bluetooth HCI Command - LE Set Extended Advertising Data
Command Opcode: LE Set Extended Advertising Data (0x2037)
Parameter Total Length: 80
Advertising Handle: 0x02
Data Operation: Complete scan response data (0x03)
Fragment Preference: The Controller should not fragment or should minimize fragmentation of Host data (0x01)
Data Length: 76
Advertising Data
Flags
Length: 2
Type: Flags (0x01)
000. .... = Reserved: 0x0
...1 .... = Simultaneous LE and BR/EDR to Same Device Capable (Host): true (0x1)
.... 1... = Simultaneous LE and BR/EDR to Same Device Capable (Controller): true (0x1)
.... .0.. = BR/EDR Not Supported: false (0x0)
.... ..1. = LE General Discoverable Mode: true (0x1)
.... ...0 = LE Limited Discoverable Mode: false (0x0)
16-bit Service Class UUIDs
Length: 3
Type: 16-bit Service Class UUIDs (0x03)
UUID 16: Device Information (0x180a)
128-bit Service Class UUIDs
Length: 17
Type: 128-bit Service Class UUIDs (0x07)
Custom UUID: caecface-e1d9-11e6-bf01-fe55135034f0 (Unknown)
Service Data - 128 bit UUID
Length: 50
Type: Service Data - 128 bit UUID (0x21)
Custom UUID: caecface-e1d9-11e6-bf01-fe55135034f0 (Unknown)
Service Data: ffff0000000000000000000000000000004142434445464748494a4b4c4d4e4f50
[Response in frame: 500]
[Command-Response Delta: 1.889ms]
Frame 500: 7 bytes on wire (56 bits), 7 bytes captured (56 bits) on interface TCP#127.0.0.1:24352, id 0
Bluetooth
Bluetooth HCI H4
Bluetooth HCI Event - Command Complete
Event Code: Command Complete (0x0e)
Parameter Total Length: 4
Number of Allowed Command Packets: 1
Command Opcode: LE Set Extended Advertising Data (0x2037)
0010 00.. .... .... = Opcode Group Field: LE Controller Commands (0x08)
.... ..00 0011 0111 = Opcode Command Field: LE Set Extended Advertising Data (0x037)
Status: Success (0x00)
[Command in frame: 499]
[Command-Response Delta: 1.889ms]
I've also tried using the exact code verbatim in this example and it has the exact same problem:
https://github.com/ProH4Ck/treadmill-bridge/blob/98e683e2380178319972af522d9251f44350a448/src/TreadmillBridge/Services/VirtualTreadmill/VirtualTreadmillService.cs#L90
Does anyone know what the problem might be?
While working on my bootloader project, I noticed that the resulting binary file is larger than the sum of the sizes of each section in the original ELF file.
After linking the bootloader image via ld, the ELF file is structured as follows:
$ readelf -S elfboot.elf
There are 10 section headers, starting at offset 0xa708:
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] .text PROGBITS 00007e00 000e00 004bc7 00 AX 0 0 16
[ 2] .rodata PROGBITS 0000c9d0 0059d0 000638 00 A 0 0 32
[ 3] .initcalls PROGBITS 0000ec20 007c20 00000c 00 WA 0 0 4
[ 4] .exitcalls PROGBITS 0000ec30 007c30 00000c 00 WA 0 0 4
[ 5] .data PROGBITS 0000ec40 007c40 0001e0 00 WA 0 0 32
[ 6] .bss NOBITS 0000ee20 007e20 00025c 00 WA 0 0 32
[ 7] .symtab SYMTAB 00000000 007e20 0016e0 10 8 166 4
[ 8] .strtab STRTAB 00000000 009500 0011bb 00 0 0 1
[ 9] .shstrtab STRTAB 00000000 00a6bb 00004a 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
p (processor specific)
I noticed that there is a rather big gap (~7KB) between the sections .rodata and .initcalls. I also checked out the content of each section:
$ objdump -s elfboot.elf
[...]
Contents of section .rodata:
[...]
cfc0 23cf0000 80000000 2fcf0000 00010000 #......./.......
cfd0 3bcf0000 00020000 47cf0000 00040000 ;.......G.......
cfe0 54cf0000 00080000 61cf0000 00100000 T.......a.......
cff0 6ecf0000 00200000 7bcf0000 00400000 n.... ..{....#..
d000 89cf0000 00800000 ........
Contents of section .initcalls:
ec20 02b50000 6db50000 feac0000 ....m.......
Contents of section .exitcalls:
[...]
My linker script tells to align every section at a 16 byte (0x10) boundary. After
objcopy -O binary elfboot.elf elfboot.bin
the resulting binary contains a huge chunk of zero filled bytes at offset 0x5200 (0xd000 - 0x7e00) all the way to offset 0x6e20 (0xec20 - 0x7e00). Now that I know where the zero filled bytes come from, how do I remove them? For reference I added the verbose output of the linker step including the linker script used:
i686-elfboot-gcc -o elfboot.elf -T elfboot.ld ./src/arch/x86/bootstrap.o ./src/arch/x86/a20.o ./src/arch/x86/bda.o ./src/arch/x86/bios.o ./src/arch/x86/copy.o ./src/arch/x86/e820.o ./src/arch/x86/entry.o ./src/arch/x86/idt.o ./src/arch/x86/realmode_jmp.o ./src/arch/x86/setup.o ./src/arch/x86/opmode.o ./src/arch/x86/pic.o ./src/arch/x86/ptrace.o ./src/arch/x86/video.o ./src/core/bdev.o ./src/core/cdev.o ./src/core/elf.o ./src/core/input.o ./src/core/interrupt.o ./src/core/loader.o ./src/core/main.o ./src/core/module.o ./src/core/pci.o ./src/core/printf.o ./src/core/string.o ./src/core/symbol.o ./src/crypto/crc32.o ./src/drivers/ide/ide.o ./src/fs/fs.o ./src/fs/file.o ./src/fs/super.o ./src/fs/ramfs/ramfs.o ./src/fs/isofs/isofs.o ./src/lib/ata/libata.o ./src/lib/tmg/libtmg.o ./src/mm/memblock.o ./src/mm/page_alloc.o ./src/mm/slub.o ./src/mm/util.o -O2 -Wl,--verbose -nostdlib -lgcc
GNU ld (GNU Binutils) 2.31.1
Supported emulations:
elf_i386
elf_iamcu
opened script file elfboot.ld
using external linker script:
==================================================
OUTPUT_FORMAT(elf32-i386)
ENTRY(_arch_start)
SECTIONS
{
/*
* The boot stack starts at 0x6000 and grows towards lower addresses.
* The stack is designed to be only one page in size, which should be
* sufficient for the bootstrap stage.
*/
. = 0x6000;
__stack_start = .;
/*
* Buffer for reading files from the boot device. Usually boot devices
* are designed to read data in chunks of 0x200 (512) or 0x800 (2048)
* bytes. The buffer is large enough to read 4 512 or 2 2048 chunks of
* data from the disk.
*/
__buffer_start = .;
. = 0x7000;
__buffer_end = .;
/*
* Actual bootstrap stage code and data.
*/
. = 0x7E00;
__bootstrap_start = .;
.text ALIGN(0x10) : {
__text_start = .;
*(.text*)
__text_end = .;
}
.rodata ALIGN(0x10) : {
__rodata_start = .;
*(.rodata*)
__rodata_end = .;
}
/*
* This section is dedicated to all built-in modules. If a module will
* be included in the elfboot binary, the module_init function pointer
* of that module is placed in this section.
*
* Make sure to initialize built-in filesystems first as we need it to
* setup the root file system node.
*/
.initcalls ALIGN(0x10) : {
__initcalls_vfs_start = .;
/* Built-in file systems */
*(.initcalls_vfs*)
__initcalls_vfs_end = .;
__initcalls_dev_start = .;
/* Built-in devices */
*(.initcalls_dev*)
__initcalls_dev_end = .;
__initcalls_start = .;
/* Built-in modules */
*(.initcalls*)
__initcalls_end = .;
}
.exitcalls ALIGN(0x10) : {
__exitcalls_start = .;
*(.exitcalls*)
__exitcalls_end = .;
}
.data ALIGN(0x10) : {
__data_start = .;
*(.data*)
__data_end = .;
}
.bss ALIGN(0x10) : {
__bss_start = .;
*(.bss*)
__bss_end = .;
}
__bootstrap_end = .;
}
==================================================
attempt to open ./src/arch/x86/bootstrap.o succeeded
./src/arch/x86/bootstrap.o
attempt to open ./src/arch/x86/a20.o succeeded
./src/arch/x86/a20.o
attempt to open ./src/arch/x86/bda.o succeeded
./src/arch/x86/bda.o
attempt to open ./src/arch/x86/bios.o succeeded
./src/arch/x86/bios.o
attempt to open ./src/arch/x86/copy.o succeeded
./src/arch/x86/copy.o
attempt to open ./src/arch/x86/e820.o succeeded
./src/arch/x86/e820.o
attempt to open ./src/arch/x86/entry.o succeeded
./src/arch/x86/entry.o
attempt to open ./src/arch/x86/idt.o succeeded
./src/arch/x86/idt.o
attempt to open ./src/arch/x86/realmode_jmp.o succeeded
./src/arch/x86/realmode_jmp.o
attempt to open ./src/arch/x86/setup.o succeeded
./src/arch/x86/setup.o
attempt to open ./src/arch/x86/opmode.o succeeded
./src/arch/x86/opmode.o
attempt to open ./src/arch/x86/pic.o succeeded
./src/arch/x86/pic.o
attempt to open ./src/arch/x86/ptrace.o succeeded
./src/arch/x86/ptrace.o
attempt to open ./src/arch/x86/video.o succeeded
./src/arch/x86/video.o
attempt to open ./src/core/bdev.o succeeded
./src/core/bdev.o
attempt to open ./src/core/cdev.o succeeded
./src/core/cdev.o
attempt to open ./src/core/elf.o succeeded
./src/core/elf.o
attempt to open ./src/core/input.o succeeded
./src/core/input.o
attempt to open ./src/core/interrupt.o succeeded
./src/core/interrupt.o
attempt to open ./src/core/loader.o succeeded
./src/core/loader.o
attempt to open ./src/core/main.o succeeded
./src/core/main.o
attempt to open ./src/core/module.o succeeded
./src/core/module.o
attempt to open ./src/core/pci.o succeeded
./src/core/pci.o
attempt to open ./src/core/printf.o succeeded
./src/core/printf.o
attempt to open ./src/core/string.o succeeded
./src/core/string.o
attempt to open ./src/core/symbol.o succeeded
./src/core/symbol.o
attempt to open ./src/crypto/crc32.o succeeded
./src/crypto/crc32.o
attempt to open ./src/drivers/ide/ide.o succeeded
./src/drivers/ide/ide.o
attempt to open ./src/fs/fs.o succeeded
./src/fs/fs.o
attempt to open ./src/fs/file.o succeeded
./src/fs/file.o
attempt to open ./src/fs/super.o succeeded
./src/fs/super.o
attempt to open ./src/fs/ramfs/ramfs.o succeeded
./src/fs/ramfs/ramfs.o
attempt to open ./src/fs/isofs/isofs.o succeeded
./src/fs/isofs/isofs.o
attempt to open ./src/lib/ata/libata.o succeeded
./src/lib/ata/libata.o
attempt to open ./src/lib/tmg/libtmg.o succeeded
./src/lib/tmg/libtmg.o
attempt to open ./src/mm/memblock.o succeeded
./src/mm/memblock.o
attempt to open ./src/mm/page_alloc.o succeeded
./src/mm/page_alloc.o
attempt to open ./src/mm/slub.o succeeded
./src/mm/slub.o
attempt to open ./src/mm/util.o succeeded
./src/mm/util.o
attempt to open /home/croemheld/Repositories/elfboot/elfboot-toolchain/lib/gcc/i686-elfboot/8.2.0/libgcc.so failed
attempt to open /home/croemheld/Repositories/elfboot/elfboot-toolchain/lib/gcc/i686-elfboot/8.2.0/libgcc.a succeeded
(/home/croemheld/Repositories/elfboot/elfboot-toolchain/lib/gcc/i686-elfboot/8.2.0/libgcc.a)_umoddi3.o
(/home/croemheld/Repositories/elfboot/elfboot-toolchain/lib/gcc/i686-elfboot/8.2.0/libgcc.a)_udivmoddi4.o
Please note that I am not able to load the sections individually into memory, since my first stage bootloader (boot sector) simply loads the entire file at a specified offset from the boot device. To reduce the size of the second stage bootloader image, I want to try to remove the gap at link time or also if possible at all, at post link time (objcopy, ...).
I've been tasked to replace C++ code to Go and I'm quite new to the Go APIs. I am using gob for encoding hundreds of key/value entries to disk pages but the gob encoding has too much bloat that's not needed.
package main
import (
"bytes"
"encoding/gob"
"fmt"
)
type Entry struct {
Key string
Val string
}
func main() {
var buf bytes.Buffer
enc := gob.NewEncoder(&buf)
e := Entry { "k1", "v1" }
enc.Encode(e)
fmt.Println(buf.Bytes())
}
This produces a lot of bloat that I don't need:
[35 255 129 3 1 1 5 69 110 116 114 121 1 255 130 0 1 2 1 3 75 101 121 1 12 0 1 3 86 97 108 1 12 0 0 0 11 255 130 1 2 107 49 1 2 118 49 0]
I want to serialize each string's len followed by the raw bytes like:
[0 0 0 2 107 49 0 0 0 2 118 49]
I am saving millions of entries so the additional bloat in the encoding increases the file size by roughly x10.
How can I serialize it to the latter without manual coding?
If you zip a file named a.txt containing the text "hello" (which is 5 characters), the result zip will be around 115 bytes. Does this mean the zip format is not efficient to compress text files? Certainly not. There is an overhead. If the file contains "hello" a hundred times (500 bytes), zipping it will result in a file being 120 bytes! 1x"hello" => 115 bytes, 100x"hello" => 120 bytes! We added 495 byes, and yet the compressed size only increased by 5 bytes.
Something similar is happening with the encoding/gob package:
The implementation compiles a custom codec for each data type in the stream and is most efficient when a single Encoder is used to transmit a stream of values, amortizing the cost of compilation.
When you "first" serialize a value of a type, the definition of the type also has to be included / transmitted, so the decoder can properly interpret and decode the stream:
A stream of gobs is self-describing. Each data item in the stream is preceded by a specification of its type, expressed in terms of a small set of predefined types.
Let's return to your example:
var buf bytes.Buffer
enc := gob.NewEncoder(&buf)
e := Entry{"k1", "v1"}
enc.Encode(e)
fmt.Println(buf.Len())
It prints:
48
Now let's encode a few more of the same type:
enc.Encode(e)
fmt.Println(buf.Len())
enc.Encode(e)
fmt.Println(buf.Len())
Now the output is:
60
72
Try it on the Go Playground.
Analyzing the results:
Additional values of the same Entry type only cost 12 bytes, while the first is 48 bytes because the type definition is also included (which is ~26 bytes), but that is a one-time overhead.
So basically you transmit 2 strings: "k1" and "v1" which are 4 bytes, and the length of strings also has to be included, using 4 bytes (size of int on 32-bit architectures) gives you the 12 bytes, which is the "minimum". (Yes, you could use a smaller type for length, but that would have its limitations. A variable-length encoding would be a better choice for small numbers, see encoding/binary package.)
All in all, encoding/gob does a pretty good job for your needs. Don't get fooled by initial impressions.
If this 12 bytes for one Entry is too "much" for you, you can always wrap the stream into a compress/flate or compress/gzip writer to further reduce the size (in exchange for slower encoding/decoding and slightly higher memory requirement for the process).
Demonstration:
Let's test the following 5 solutions:
Using a "naked" output (no compression)
Using compress/flate to compress the output of encoding/gob
Using compress/zlib to compress the output of encoding/gob
Using compress/gzip to compress the output of encoding/gob
Using github.com/dsnet/compress/bzip2 to compress the output of encoding/gob
We will write a thousand entries, changing keys and values of each, being "k000", "v000", "k001", "v001" etc. This means the uncompressed size of an Entry is 4 byte + 4 byte + 4 byte + 4 byte = 16 bytes (2x4 bytes text, 2x4 byte lengths).
The code looks like this:
for _, name := range []string{"Naked", "flate", "zlib", "gzip", "bzip2"} {
buf := &bytes.Buffer{}
var out io.Writer
switch name {
case "Naked":
out = buf
case "flate":
out, _ = flate.NewWriter(buf, flate.DefaultCompression)
case "zlib":
out, _ = zlib.NewWriterLevel(buf, zlib.DefaultCompression)
case "gzip":
out = gzip.NewWriter(buf)
case "bzip2":
out, _ = bzip2.NewWriter(buf, nil)
}
enc := gob.NewEncoder(out)
e := Entry{}
for i := 0; i < 1000; i++ {
e.Key = fmt.Sprintf("k%3d", i)
e.Val = fmt.Sprintf("v%3d", i)
enc.Encode(e)
}
if c, ok := out.(io.Closer); ok {
c.Close()
}
fmt.Printf("[%5s] Length: %5d, average: %5.2f / Entry\n",
name, buf.Len(), float64(buf.Len())/1000)
}
Output:
[Naked] Length: 16036, average: 16.04 / Entry
[flate] Length: 4120, average: 4.12 / Entry
[ zlib] Length: 4126, average: 4.13 / Entry
[ gzip] Length: 4138, average: 4.14 / Entry
[bzip2] Length: 2042, average: 2.04 / Entry
Try it on the Go Playground.
As you can see: the "naked" output is 16.04 bytes/Entry, just little over the calculated size (due to the one-time tiny overhead discussed above).
When you use flate, zlib or gzip to compress the output, you can reduce the output size to about 4.13 bytes/Entry, which is about ~26% of the theoretical size, I'm sure that satisfies you. If not, you can reach out to libs providing compression with higher efficiency like bzip2, which in the above example resulted in 2.04 bytes/Entry, being 12.7% of the theoretical size!
(Note that with "real-life" data the compression ratio would probably be a lot higher as the keys and values I used in the test are very similar and thus really well compressible; still ratio should be around 50% with real-life data).
Use protobuf to efficiently encode your data.
https://github.com/golang/protobuf
Your main would look like this:
package main
import (
"fmt"
"log"
"github.com/golang/protobuf/proto"
)
func main() {
e := &Entry{
Key: proto.String("k1"),
Val: proto.String("v1"),
}
data, err := proto.Marshal(e)
if err != nil {
log.Fatal("marshaling error: ", err)
}
fmt.Println(data)
}
You create a file, example.proto like this:
package main;
message Entry {
required string Key = 1;
required string Val = 2;
}
You generate the go code from the proto file by running:
$ protoc --go_out=. *.proto
You can examine the generated file, if you wish.
You can run and see the results output:
$ go run *.go
[10 2 107 49 18 2 118 49]
"Manual coding", you're so afraid of, is trivially done in Go using the standard encoding/binary package.
You appear to store string length values as 32-bit integers in big-endian format, so you can just go on and do just that in Go:
package main
import (
"bytes"
"encoding/binary"
"fmt"
"io"
)
func encode(w io.Writer, s string) (n int, err error) {
var hdr [4]byte
binary.BigEndian.PutUint32(hdr[:], uint32(len(s)))
n, err = w.Write(hdr[:])
if err != nil {
return
}
n2, err := io.WriteString(w, s)
n += n2
return
}
func main() {
var buf bytes.Buffer
for _, s := range []string{
"ab",
"cd",
"de",
} {
_, err := encode(&buf, s)
if err != nil {
panic(err)
}
}
fmt.Printf("%v\n", buf.Bytes())
}
Playground link.
Note that in this example I'm writing to a byte buffer, but that's for demonstration purposes only—since encode() writes to an io.Writer, you can pass it an opened file, a network socket and anything else implementing that interface.
I'm using GStreamer and sending audio using this pipeline:
gst-launch-1.0 -v filesrc location=soundfile.mp3 ! mad ! audioconvert ! audio/x-raw, layout=interleaved, format=F32LE, channels=2 ! udpsink blocksize=512 port=5005 host=127.0.0.1
However, blocksize doesn't appear to be working at all. This is the doc for udpsink, accessed by gst-inspect udpsink:
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "udpsink0"
preroll-queue-len : Number of buffers to queue during preroll
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 0
sync : Sync on the clock
flags: readable, writable
Boolean. Default: true
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: -1
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: false
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
enable-last-buffer : Enable the last-buffer property
flags: readable, writable
Boolean. Default: true
last-buffer : The last buffer received in the sink
flags: readable
MiniObject of type "GstBuffer"
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
throttle-time : The time to keep between rendered buffers (unused)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
bytes-to-serve : Number of bytes received to serve to clients
flags: readable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
bytes-served : Total number of bytes sent to all clients
flags: readable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
sockfd : Socket to use for UDP sending. (-1 == allocate)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
closefd : Close sockfd if passed as property on state change
flags: readable, writable
Boolean. Default: true
sock : Socket currently in use for UDP sending. (-1 == no socket)
flags: readable
Integer. Range: -1 - 2147483647 Default: -1
clients : A comma separated list of host:port pairs with destinations
flags: readable, writable
String. Default: "localhost:4951"
auto-multicast : Automatically join/leave the multicast groups, FALSE means user has to do it himself
flags: readable, writable
Boolean. Default: true
ttl : Used for setting the unicast TTL parameter
flags: readable, writable
Integer. Range: 0 - 255 Default: 64
ttl-mc : Used for setting the multicast TTL parameter
flags: readable, writable
Integer. Range: 0 - 255 Default: 1
loop : Used for setting the multicast loop parameter. TRUE = enable, FALSE = disable
flags: readable, writable
Boolean. Default: true
qos-dscp : Quality of Service, differentiated services code point (-1 default)
flags: readable, writable
Integer. Range: -1 - 63 Default: -1
send-duplicates : When a distination/port pair is added multiple times, send packets multiple times as well
flags: readable, writable
Boolean. Default: true
buffer-size : Size of the kernel send buffer in bytes, 0=default
flags: readable, writable
Integer. Range: 0 - 2147483647 Default: 0
host : The host/IP/Multicast group to send the packets to
flags: readable, writable
String. Default: "localhost"
port : The port to send the packets to
flags: readable, writable
Integer. Range: 0 - 65535 Default: 4951
This is already somewhat confusing, as the default value for blocksize is listed as both 0 and 4096. It seems that it is 4096, however, as that is my UDP packet size no matter what value I use for blocksize. What's more confusing is that I can scarcely find any mention of the blocksize property anywhere online, even in GStreamer's own documentation: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-udpsink.html
The only properties mentioned are host and port. Has blocksize been deprecated or something? And if so, is there any way to control the amount of data sent in each UDP packet? I've tried using the mtu property in RTP with no luck (see here: gstreamer RTP packet size) and am kind of at my wits' end with this.
When you are using gstinspect on an element, it shows you all properties - not only from a particular element but also from its parents. Which is truly expected behaviour.
UdpSink indeed has only host and port property. Parent - GstMultiUDPSink has more.
You can find an inheritance chain in GStreamer documentation in form of a tree (Object Hierarchy paragraph). Unfortunately, not all links are working correctly, so it is better to google proper link for the parent element.
Ok, back to question, blocksize you are mentioning is property of GstBaseSink
and is described as:
The amount of bytes to pull when operating in pull mode.
Flags: Read / Write
Default value: 4096
So it defines how much data it will pull from upstream element. It will not affect UDP packets. Probably what you are looking for is buffer-size property of GstMultiUDPSink:
“buffer-size” gint
Size of the kernel send buffer in bytes, 0=default.
Flags: Read / Write
Allowed values: >= 0
Default value: 0
I have an ARM-based device running Embedded Linux and I have observed that when I use the C library's system() call, the return code is incorrect. Here is a test program that demonstrates the behavior:
#include <stdlib.h>
#include <stdio.h>
int main(void)
{
int ret = system("exit 42");
printf("Should return 42 for system() call: %d\n", ret);
printf("Returning 43 to shell..\n");
exit(43);
};
And here is the program output on the device:
# returnCodeTest
Should return 42 for system() call: 10752
Returning 43 to shell..
The value "10752" is returned by system() instead of "42". 10752 is 42 when left-shifted by 8:
Python 2.7.3 (default, Feb 27 2014, 20:00:17)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 42<<8
10752
So is suspect one of the following is going on somewhere:
The byte order is getting swapped
The value is getting shifted by 8 bits
Incompatible struct definitions are being used
When I run strace I see the following:
# strace /usr/bin/returnCodeTest
...
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x4001e308) = 977
wait4(977, [{WIFEXITED(s) && WEXITSTATUS(s) == 42}], 0, NULL) = 977
rt_sigaction(SIGINT, {SIG_DFL, [], 0x4000000 /* SA_??? */}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], 0x4000000 /* SA_??? */}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=977, si_status=42, si_utime=0, si_stime=0} ---
fstat64(1, {st_mode=S_IFCHR|0622, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001f000
write(1, "Should return 42 for system() ca"..., 42Should return 42 for system() call: 10752
) = 42
write(1, "Returning 43 to shell..\n", 24Returning 43 to shell..
) = 24
exit_group(43) = ?
+++ exited with 43 +++
wait4() returns with the correct status (si_status=42), but when it gets printed to standard output the value is shifted by 8 bits, and it looks like it is happening in a library. Interestingly the write returns a value of 42. I wonder if this is a hint as to what is going on...
Unfortunately I cannot get ltrace to compile and run on the device. Has anyone seen this type of behavior before or have any ideas (possibly architecture-specific) on where to look?
$man 3 system
Return Value
The value returned is -1 on error (e.g., fork(2) failed), and the
return status of the command otherwise. This latter return status is
in the format specified in wait(2). Thus, the exit code of the command
will be WEXITSTATUS(status).
$man 2 wait
WEXITSTATUS(status) returns the exit status of the child. This
consists of the least significant 8 bits of the status argument that
the child specified in a call to exit(3) or _exit(2) or as the
argument for a return statement in main(). This macro should only be
employed if WIFEXITED returned true.
I think exit codes are different from return values and is specific to OS.
Linux does the following when you call "exit" with a code.
(error_code&0xff)<<8
SYSCALL_DEFINE1(exit, int, error_code)
{
do_exit((error_code&0xff)<<8);
}
Take look at the below link (exit status codes in Linux).
Are there any standard exit status codes in Linux?