I have written following fbs file
namespace testing;
table polygon {
x : double;
y : double;
}
table layer {
layer_name : string;
polygons : [polygon];
}
root_type layer;
My plan is to serialize approx 5 Million coordinates and dump it into one file. Problem is what I see is the number of bytes is increased compared to what I was expecting. I am expecting it should be arounf (5M* 16) bytes. But the size what I am getting is 140000032 bytes
Here is the java code which I am using for dumping serialize data into a file.
FlatBufferBuilder fbb = new FlatBufferBuilder(0);
String s = "Product1";
int str = fbb.createString("layer1");
int size = 1 * 5 * 1000000;
int[] offset = new int[size];
int cur = 0;
for (double i = 0; i < size; i++) {
fbb.startTable(2);
polygon.addX(fbb, i);
polygon.addY(fbb, i);
offset[cur++] = polygon.endpolygon(fbb);
}
int arrayoffset = layer.createPolygonsVector(fbb, offset);
layer.startlayer(fbb);
layer.addLayerName(fbb, str);
layer.addPolygons(fbb, arrayoffset);
int bla = layer.endlayer(fbb);
fbb.finish(bla);
ByteBuffer bf = fbb.dataBuffer().duplicate();
File myfile = new File("/tmp/test.dat");
try {
FileChannel channel = new FileOutputStream(myfile).getChannel();
channel.write(bf);
channel.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
Please let me know if I am doing something wrong here
change table polygon into struct polygon. table is extensible and has some fixed overhead per element, and also is referred to by the vector over an offset. A struct is not extensible (which seems fine for an xy pair), and has to be serialized inline (see example in the tutorial), and will give you the size you expect.
Related
As part of my final year project, I am testing the Bouncycastle library on SHA-3.
I have found the source code to calculate the hash value of a string:
String input = "hello" ;
SHA3.DigestSHA3 digestSHA3 = new SHA3.Digest256();
byte[] digest = digestSHA3.digest(input.getBytes());
System.out.println("SHA3-256 = " + Hex.toHexString(digest));
but i want to calculate the hash value of a matrix, Anyone who can help me with this?
You need to uniquely convert matrix to byte array. One of the possible solutions:
private static byte[] intToBytes(int value) {
return new byte[] {
(byte)(value >>> 24),
(byte)(value >>> 16),
(byte)(value >>> 8),
(byte)value
};
}
public static void main(String[] args) throws Exception {
int[][] matrix = new int[3][5];
SHA3.DigestSHA3 sha3 = new SHA3.Digest256();
int height = matrix.length;
int width = matrix[0].length;
sha3.update(intToBytes(height)); // add height of the matrix
sha3.update(intToBytes(width)); // add width
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
sha3.update(intToBytes(matrix[i][j])); // add all values
}
}
byte[] digest = digest.digest();
}
On my Arduino Mega 2560, I'm trying to run a motor that turns a 20-vial container (accepting int input 1-20) while regulating temperature via PID of a separate cooler. I am generally new to this field of technology so bear with me. I also have an interrupt set up for an encoder to keep track of vial position.
The void serialEvent() and void loop() are the most important portions to look at, but I decided to put the rest of the code in there just in case you needed to see it.
#include <PID_v1.h>
#include <SPI.h>
#include <TMC26XStepper.h>
#define COOL_INPUT 0
#define PIN_OUTPUT 9
TMC26XStepper tmc26XStepper = TMC26XStepper(200,5,7,6,500);
int step = 6;
int value;
int i;
char junk = ' ';
volatile long enc_count = 0;
const byte interruptPinA = 2;
const byte interruptPinB = 3;
//Define Variables we'll be connecting to
int outMax = 255;
int outMin = -145;
double Setpoint, Input, Output;
double heatInput, heatOutput, originalInput;
//Specify the links and initial tuning parameters
// AGGRESSIVE VALUES (to get to 4 deg C)
double aggKp=8.0, aggKi=3.0, aggKd=0.15;
// CONSERVATIVE VALUES (to hover around 4 deg C)
double consKp=2.5, consKi = 0.0, consKd = 1.0;
PID myPID(&Input, &Output, &Setpoint, aggKp, aggKi, aggKd, REVERSE);
void setup()
{
pinMode(step, OUTPUT);
pinMode(interruptPinA, INPUT_PULLUP);
pinMode(interruptPinB, INPUT_PULLUP);
attachInterrupt(digitalPinToInterrupt(interruptPinA), encoder_isr, CHANGE);
attachInterrupt(digitalPinToInterrupt(interruptPinB), encoder_isr, CHANGE);
//initialize the variables we're linked to
Input = (5.0*analogRead(COOL_INPUT)*100.0) / 1024;
Setpoint = 10.75;
myPID.SetOutputLimits(outMin, outMax);
//turn the PID on
myPID.SetMode(AUTOMATIC);
Serial.begin(115200);
tmc26XStepper.setSpreadCycleChopper(2,24,8,6,0);
tmc26XStepper.setMicrosteps(32);
tmc26XStepper.setStallGuardThreshold(4,0);
Serial.println("...started...");
tmc26XStepper.start();
Serial.flush();
Serial.println("Enter vial numbers 1-20");
}
void loop() {
Input = (5.0*analogRead(COOL_INPUT)*100.0) / 1024;
// A BUNCH OF CODE FOR TEMP REGULATION
Serial.println(Input);
delay(150);
}
void serialEvent() {
while (Serial.available() == 0) {}
i = Serial.parseInt();
Serial.print("position: ");
Serial.print(i);
Serial.print(" ");
while (Serial.available() > 0) {
junk = Serial.read();
}
if (i == 1) {
value = 0;
} else {
int num = i - 1;
value = num * 72;
}
while (enc_count != value) {
digitalWrite(6, HIGH);
delayMicroseconds(100);
digitalWrite(6, LOW);
delayMicroseconds(100);
if (enc_count == 1440) {
enc_count = 0;
}
}
Serial.println(enc_count);
}
// INFO FOR ENCODER
void encoder_isr() {
static int8_t lookup_table[] = {0,-1,1,0,1,0,0,-1,-1,0,0,1,0,1,-1,0};
static uint8_t enc_val = 0;
enc_val = enc_val << 2;
enc_val = enc_val | ((PIND & 0b1100) >> 2);
enc_count = enc_count + lookup_table[enc_val & 0b1111];
}
So, originally I had the two processes tested separately (vial position + encoder, then temperature regulation) and everything did exactly as it was supposed to. Now, I fused the code together and stored the vial position entry in the serialEvent() method to keep the temperature reading continuous and the vial position entry available for whenever I decided to provide input. However, when I put in a value, the program stops all together. I am able to see the number I entered (position: 5), but the Serial.println(enc_count) never gets printed. On top of the that, the temperature readings stop displaying readings.
Any thoughts? Need more information?
On certain images, when I call:
PdfImageObject pimg = new PdfImageObject(stream);
Image bmp = pimg.GetDrawingImage();
The Image that is returned is twisted. I've seen this before and it usually has to do with byte alignment but I'm not sure how to get around this.
The /DecodeParms for this object are /EndOfLine true /K 0 /Columns 3300.
I have tried using the GetStreamBytesRaw() with BitMiracle.LibTiff and with it I can get the data formatted properly although the image is rotated. I'd prefer for GetDrawingImage() to decode the data properly if possible, assuming that is the problem.
I could provide the PDF via email if requested.
Thanks,
Darren
For anyone else that runs across this scenario here is my solution. The key to this puzzle was understanding that /K 0 is G3, /K -1 (or anything less than 0) is G4 /K 1 (or anything greater than 0) is G3-2D.
The twisting happens when you try to make G3 compressed data fit into a G4 image which it appears that is what iTextSharp may be doing. I know it definitely does not work with how I have iTextSharp implemented in my project. I confess that I cannot decipher all the decoding stuff that iTextSharp is doing so it could be something I'm missing too.
EndOfLine didn't have any part in this puzzle but I still think putting line feeds in binary data is a strange practice.
99% of this code came from BitMiracle.LibTiff.Net - Thank you.
int nK = 0;// Default to 0 like the PDF Spec
PdfObject oDecodeParms = stream.Get(PdfName.DECODEPARMS);
if (oDecodeParms is PdfDictionary)
{
PdfObject oK0 = ((PdfDictionary)oDecodeParms).Get(PdfName.K);
if (oK0 != null)
nK = ((PdfNumber)oK0).IntValue;
}
using (MemoryStream ms = new MemoryStream())
{
using (Tiff tiff = Tiff.ClientOpen("custom", "w", ms, new TiffStream()))
{
tiff.SetField(TiffTag.IMAGEWIDTH, width);
tiff.SetField(TiffTag.IMAGELENGTH, height);
if (nK == 0 || nK > 0) // 0 = Group 3, > 0 = Group 3 2D
tiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX3);
else if (nK < 0) // < 0 = Group 4
tiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX4);
tiff.SetField(TiffTag.BITSPERSAMPLE, bpc);
tiff.SetField(TiffTag.SAMPLESPERPIXEL, 1);
tiff.WriteRawStrip(0, rawBytes, rawBytes.Length); //saving the tiff file using the raw bytes retrieved from the PDF.
tiff.Close();
}
TiffStreamForBytes byteStream = new TiffStreamForBytes(ms.ToArray());
using (Tiff input = Tiff.ClientOpen("bytes", "r", null, byteStream))
{
int stride = input.ScanlineSize();
Bitmap result = new Bitmap(width, height, pixelFormat);
ColorPalette palette = result.Palette;
palette.Entries[0] = System.Drawing.Color.White;
palette.Entries[1] = System.Drawing.Color.Black;
result.Palette = palette;
for (int i = 0; i < height; i++)
{
Rectangle imgRect = new Rectangle(0, i, width, 1);
BitmapData imgData = result.LockBits(imgRect, ImageLockMode.WriteOnly, pixelFormat);
byte[] buffer = new byte[stride];
input.ReadScanline(buffer, i);
System.Runtime.InteropServices.Marshal.Copy(buffer, 0, imgData.Scan0, buffer.Length);
result.UnlockBits(imgData);
}
}
}
/// <summary>
/// Custom read-only stream for byte buffer that can be used
/// with Tiff.ClientOpen method.
/// </summary>
public class TiffStreamForBytes : TiffStream
{
private byte[] m_bytes;
private int m_position;
public TiffStreamForBytes(byte[] bytes)
{
m_bytes = bytes;
m_position = 0;
}
public override int Read(object clientData, byte[] buffer, int offset, int count)
{
if ((m_position + count) > m_bytes.Length)
return -1;
Buffer.BlockCopy(m_bytes, m_position, buffer, offset, count);
m_position += count;
return count;
}
public override void Write(object clientData, byte[] buffer, int offset, int count)
{
throw new InvalidOperationException("This stream is read-only");
}
public override long Seek(object clientData, long offset, SeekOrigin origin)
{
switch (origin)
{
case SeekOrigin.Begin:
if (offset > m_bytes.Length)
return -1;
m_position = (int)offset;
return m_position;
case SeekOrigin.Current:
if ((offset + m_position) > m_bytes.Length)
return -1;
m_position += (int)offset;
return m_position;
case SeekOrigin.End:
if ((m_bytes.Length - offset) < 0)
return -1;
m_position = (int)(m_bytes.Length - offset);
return m_position;
}
return -1;
}
public override void Close(object clientData)
{
// nothing to do
return;
}
public override long Size(object clientData)
{
return m_bytes.Length;
}
}
I'm very new to Arduino and C programming.
I'm making a GPS speedo and I'm trying to read in some serial, store a value from a substring and echo it back via serial.
At the moment I'm having problems storing the substring.
I've gotten to the point where I'm able to get some data between < and >.
But the data doesn't come in like that. It's a NMEA data stream and the data I want is between ,N, and ,K,.
So I've been trying to replace ,N, with < and ,K, with > .
Just can't get it to work. I get error: request for member 'replace' in 'c', which is of non-class type 'char'
Here's my code so far....
int indata = 0;
int scrubdata = 0;
char inString[32];
int stringPos = 0;
boolean startRead = false; // is reading?
void setup() {
Serial.begin(4800);
}
void loop() {
String pageValue = readPage();
Serial.print(pageValue);
}
String readPage(){
//read the page, and capture & return everything between '<' and '>'
stringPos = 0;
memset( &inString, 0, 32 ); //clear inString memory
while(true){
if (Serial.available() > 0) {
char c = Serial.read();
c.replace(",N,", "<");
c.replace(",K,", ">");
if (c == '<' ) { //'<' is our begining character
startRead = true; //Ready to start reading the part
}
else if(startRead){
if(c != '>'){ //'>' is our ending character
inString[stringPos] = c;
stringPos ++;
}
else{
//got what we need here! We can disconnect now
startRead = false;
return inString;
}
}
}
}
}
By Default:
Serial.read() returns an int if you must process the data this way, try casting it to char with:
char c = (char) Serial.read();
Another way to do this:
Would be to seek your beginning string (discarding un-needed data) using Serial.find() then reading data until you met your end character ",K," with Serial.readBytesUntil()
Something like this would work quite well:
char inData[64]; //adjust for your data size
Serial.setTimeout(2000); //Defaults to 1000 msecs set if necessary
Serial.find(",N,"); //Start of Data
int bRead = Serial.readBytesUntil(",K,", inData, 64); //Read until end of data
inData[bRead] = 0x00; //Zero terminate if using this as a string
return inData;
I have been asked to provide a WCF service that allows a blob (potentially 1GB) to be downloaded in chunks as an offset byte[] for consumption by a Silverlight application. Essentially, the operation will have a parameter for number of bytes to offset and the max number of bytes to return, nothing complex I think.
The code I have so far is:
[OperationContract]
public byte[] Download(String url, int blobOffset, int bufferSize)
{
var blob = new CloudBlob(url);
using(var blobStream = blob.OpenRead())
{
var buffer = new byte[bufferSize];
blobStream.Seek(blobOffset, SeekOrigin.Begin);
int numBytesRead = blobStream.Read(buffer, 0, bufferSize);
if (numBytesRead != bufferSize)
{
var trimmedBuffer = new byte[numBytesRead];
Array.Copy(buffer, trimmedBuffer, numBytesRead);
return trimmedBuffer;
}
return buffer;
}
}
I have tested this (albeit with relatively small files < 2MB) and it does work, but my questions are:
Can someone suggest improvements to the code?
Is there a better approach given the requirement?
using (BlobStream blobStream = blob.OpenRead())
{
bool getSuccess = false;
int getTries = 0;
rawBytes = new byte[blobStream.Length];
blobStream.Seek(0, SeekOrigin.Begin);
int blockSize = 4194304; //Start at 4 mb per batch
int index = 0;
int documentSize = rawBytes.Length;
while (getTries <= 10 && !getSuccess)
{
try
{
int batchSize = blockSize;
while (index < documentSize)
{
if ((index + batchSize) > documentSize)
batchSize = documentSize - index;
blobStream.Read(rawBytes, index, batchSize);
index += batchSize;
}
getSuccess = true;
}
catch (Exception e)
{
if (getTries > 9)
throw e;
else
blockSize = blockSize / 2; // Reduce by half for each attempt
}
finally
{ getTries++; }
}
}
You could return the blob as a stream instead of a byte array. There is a code sample in a related question here: Returning Azure BLOB from WCF service as a Stream - Do we need to close it?
Note there are some restrictions on which bindings you can use when you return a stream.