Proto3 release notes states:
The
deterministic serialization is, however, NOT canonical across languages; it
is also unstable across different builds with schema changes due to unknown
fields. Users who need canonical serialization, e.g. persistent storage in
a canonical form, fingerprinting, etc, should define their own
canonicalization specification and implement the serializer using reflection
APIs rather than relying on this API.
What I would like to achieve is to have a deterministic serialisation of protobuf message to carry a crypto signature along with it. As I understand due to differences in serialisers binary data could differ and signature would become invalid.
package Something
message Request {
Payload payload = 1;
// signature of serialised payload
bytes signature = 2;
message Payload {
string user_id_from = 1;
uint64 amount = 2;
string user_id_to = 3;
}
}
What is the way to do this using protobuf reflection?
This doesn't answer the question directly, but may solve your issue: don't store the payload as a message, but store the serialized bytes alongside with the signature.
message Request {
// Serialized Payload message.
bytes payload = 1;
// signature of serialised payload
bytes signature = 2;
}
message Payload {
string user_id_from = 1;
uint64 amount = 2;
string user_id_to = 3;
}
This may be a little less convenient to work with in code, but has the advantage of preserving all the forwards and backwards-compatibility guarantees of protobuf.
It also frees you from serializing the message twice when writing it (once as a subfield, once to get the signature).
Related
Could someone explain how to properly use Scan() and Value() in the following example?
I'm trying to make use of the following examples:
https://github.com/jinzhu/gorm/issues/2017#issuecomment-537627961
https://github.com/travisjeffery/proto-go-sql/blob/master/_example/person_sql.go
My proto:
message Timestamp {
google.protobuf.Timestamp timestamp = 1;
}
message User {
uint32 ID = 1;
Timestamp createdAt = 2;
}
Code (need to fix time.Now()):
package v1
import (
"database/sql/driver"
"fmt"
"time"
"github.com/golang/protobuf/ptypes"
)
func (u *User) Create() {
u.CreatedAt = time.Now() // FIXME: ERROR. How do I make use of Scan() and Value() here?
// saving to SQL database
}
func (ts *Timestamp) Scan(value interface{}) error {
switch t := value.(type) {
case time.Time:
var err error
ts.Timestamp, err = ptypes.TimestampProto(t)
if err != nil {
return err
}
default:
return fmt.Errorf("Not a protobuf Timestamp")
}
return nil
}
func (ts Timestamp) Value() (driver.Value, error) {
return ptypes.Timestamp(ts.Timestamp)
}
The scanner and valuer interfaces are not really things you'll use yourself, not when it comes to storing custom types in a DB, at least. I'll first cover the use of the Scan() and Value() functions, then I'll address your issues.
When you get a sql.Row result, and want to assign (scan) the values from the result set into your variables of a custom type. The docs show the sql.Row.Scan() function takes 0 or more arguments of type interface{}, basically anything. (check docs here).
In the list of supported types into which values can be scanned, the last line is the important one:
any type implementing Scanner (see Scanner docs)
With the function func (ts *Timestamp) Scan(value interface{}) error {, the Timestamp type now implements the Scanner interface, thus allowing sql.Row to assign values to this type. The documentation for the Scanner interface is located right below the docs for Scan(), which I linked above.
Of course, that helps you to read values from the DB, gets you nowhere when it comes to storing these types. For that, you need the Valuer interface. In case you haven't guessed it yet, the func (ts Timestamp) Value() (driver.Value, error) function indeed makes it so your Timestamp type implements this interface. The documentation for the driver.Valuer interface can be found here, all the way at the bottom.
The point of the Valuer interface is to allow a way to convert any type to a driver.Value, that the driver can work with and store in the DB (again: docs here).
Fixing the issues
First up, I'm goign to have to assume your protoc output is written to the v1 package. If it isn't, it's not going to work very well for you.
The offending line is indeed the one you marked out:
u.CreatedAt = time.Now()
First up, User.CreatedAt is of type Timestamp, which is in and of itself a message containing a single timestamp. To set the CreatedAt time to time.Now(), you need to initialise the CreatedAt field correctly:
u.CreatedAt = &Timestamp{
Timestamp: ptypes.TimestampNow(), // this returns a PROTOBUF TYPE!
}
You are doing this in your Scan and Value functions already, so I really don't get why you didn't do it here...
Advice
If the protoc output is indeed written to the v1 package, I really, really would remove the User.Create() function. In fact, I'd outright kill it. Your protocol buffers are used for communication. Exposing your program via RPC's. It's an API. These message types are essentially request and response objects (glorified DTO's if you will). You're adding this Create function to it, which turns them into AR types. It makes your protobuf package unusable. The beauty of gRPC is that you generate golang, C++, Python, ... code that others can use to call your program. If you make your gRPC package dependent on a database, as you are doing, I personally would never, ever use it.
I'm parsing a FlatBuffers binary file and create POJOs from the data (Plain Old Javascript Objects) for some reason.
In my .fbs file I have for example a geo_id defined as follows:
table Properties {
geo_id:long;
tags:[TagIndex];
}
In the javascript in my HTML I create a POJO feature object lie this:
function createFeature(rawFeature, cell) {
var feature = {
id: rawFeature.properties().geoId(),
geometry: null,
properties: {}
}
return feature;
}
My expectation was that I get a plain number (long), but I'm getting an object with "low" and "high" where "low" seems to be the id. Though I'm a bit confused and would like to know the proper way to convert this data to plain old variables.
A long is a 64-bit number which can't be represented natively in JS. To preserve the information, it is represented as 2 32-bit numbers.
If you are really using all the bits in a long, then there's no way to convert it to a single number safely. If the high bits are all 0, then you could use just the low bits to represent it as a single number in JS.
You may want to use int in your schema for convenience if you don't need the high bits.
I send the following message with content type application/json:
However whene i get messages from the same RabbitMQ Web console, it shows the payload as String.
What am I doing wrong? Or am I fundamentally misunderstanding and the Payload is always of type String?
From the official docs:
AMQP messages also have a payload (the data that they carry), which AMQP brokers treat as an opaque byte array. The broker will not inspect or modify the payload. It is possible for messages to contain only attributes and no payload. It is common to use serialisation formats like JSON, Thrift, Protocol Buffers and MessagePack to serialize structured data in order to publish it as the message payload. AMQP peers typically use the "content-type" and "content-encoding" fields to communicate this information, but this is by convention only.
So basically, RabbitMQ has no knowledge on JSON, messages all are just byte arrays to it
From NodeJS Context:
If we want to send JSON object as message, we may get the following error:
The first argument must be of type string or an instance of Buffer,
ArrayBuffer, or Array or an Array-like Object. Received an instance of
Object
So, we can convert the JSON payload as string and parse it in the worker. We stringify the JSON object before sending the data the Queue-
let payloadAsString = JSON.stringify(payload);
And from worker's end, we can then JSON.parse
let payload = JSON.parse(msg.content.toString());
//then access the object as we normally do, i.e. :
let id = payload.id;
For anyone using .Net to send objects via RabbitMQ.
You have to serialise your JSON object to byte array, send via RabbitMQ then de-serialise after receiving. You can do this like this:
Install the Newtonsoft JSON library
using Newtonsoft.Json;
Create a model for your JSON object message (in this case AccountMessage)
Serialise your object into byte array like this:
byte[] messagebuffer = Encoding.Default.GetBytes(JsonConvert.SerializeObject(accountMessage) );
After receiving the message data, you can de-serialise like this:
AccountMessage receivedMessage = JsonConvert.DeserializeObject<AccountMessage>(Encoding.UTF8.GetString(body));
from here
Content Type and Encoding
The content (MIME media) type and content encoding fields allow publishers communicate how message payload should be deserialized and decoded by consumers.
RabbitMQ does not validate or use these fields, it exists for applications and plugins to use and interpret.
by the way, using the rabbitMQ web gui, you use the words content_type, however in code (javascript confirmed), you use the key name contentType. it's a subtle difference, but enough to drive you crazy.
I am currently sending data between my PC and an ARM M4 Microcontroller via UART. I've defined my own protocol where each message looks like this:
[START_CHAR LEN TYPE SEQ DATA CRC]
The START_CHAR and LEN fields help me determine when the data ends, after which I look up the TYPE (constant offset of 3) to figure out what data came in order to unpack it into a message class.
Now I'm looking into flatbuffers and it seems perfect except that I cannot encode the TYPE into the message without including it inside the actual message. Here is what I am trying to do:
namespace FlatMessage;
uint8 const TYPE = 50; // does not compile
table String {
value:string;
}
root_type String;
I could create an Enum but that is messy. Thank you!
[EDIT] I should add that I could just change the protocol to have an END_CHAR but I need to support the TYPE field for legacy reasons.
Well actually, I suppose I would still need the type to figure out how to deserialize it as a flatbuffer.
e.g.
uint8_t *buf = builder.GetBufferPointer(); // I can do this with END_CHAR because I could get the buffer.
auto receive_string = GetString(buf); // But I wouldn't know what the type is. e.g. this could be GetCoolString(buf).
You have a couple of options to store a type with a FlatBuffer:
Prefix a buffer yourself with a type.
Use the file_identifier feature of FlatBuffers, to make it possible to identify the type of FlatBuffer.
Store the type in FlatBuffers itself, by using a union type. Make the root table have a single union field.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
So after CodingHorror's fun with encryption and the thrashing comments, we are reconsidering doing our own encryption.
In this case, we need to pass some information that identifies a user to a 3rd party service which will then call back to a service on our website with the information plus a hash.
The 2nd service looks up info on that user and then passes it back to the 3rd party service.
We want to encrypt this user information going into the 3rd party service and decrypt it after it comes out. So it is not a long lived encryption.
On the coding horror article, Coda Hale recommended BouncyCastle and a high level abstraction in the library to do the encryption specific to a particular need.
My problem is that the BouncyCastle namespaces are huge and the documentation is non-existant. Can anyone point me to this high level abstraction library? (Or another option besides BouncyCastle?)
High level abstraction? I suppose the highest level abstractions in the Bouncy Castle library would include:
The BlockCipher interface (for symmetric ciphers)
The BufferedBlockCipher class
The AsymmetricBlockCipher interface
The BufferedAsymmetricBlockCipher class
The CipherParameters interface (for initializing the block ciphers and asymmetric block ciphers)
I am mostly familiar with the Java version of the library. Perhaps this code snippet will offer you a high enough abstraction for your purposes (example is using AES-256 encryption):
public byte[] encryptAES256(byte[] input, byte[] key) throws InvalidCipherTextException {
assert key.length == 32; // 32 bytes == 256 bits
CipherParameters cipherParameters = new KeyParameter(key);
/*
* A full list of BlockCiphers can be found at http://www.bouncycastle.org/docs/docs1.6/org/bouncycastle/crypto/BlockCipher.html
*/
BlockCipher blockCipher = new AESEngine();
/*
* Paddings available (http://www.bouncycastle.org/docs/docs1.6/org/bouncycastle/crypto/paddings/BlockCipherPadding.html):
* - ISO10126d2Padding
* - ISO7816d4Padding
* - PKCS7Padding
* - TBCPadding
* - X923Padding
* - ZeroBytePadding
*/
BlockCipherPadding blockCipherPadding = new ZeroBytePadding();
BufferedBlockCipher bufferedBlockCipher = new PaddedBufferedBlockCipher(blockCipher, blockCipherPadding);
return encrypt(input, bufferedBlockCipher, cipherParameters);
}
public byte[] encrypt(byte[] input, BufferedBlockCipher bufferedBlockCipher, CipherParameters cipherParameters) throws InvalidCipherTextException {
boolean forEncryption = true;
return process(input, bufferedBlockCipher, cipherParameters, forEncryption);
}
public byte[] decrypt(byte[] input, BufferedBlockCipher bufferedBlockCipher, CipherParameters cipherParameters) throws InvalidCipherTextException {
boolean forEncryption = false;
return process(input, bufferedBlockCipher, cipherParameters, forEncryption);
}
public byte[] process(byte[] input, BufferedBlockCipher bufferedBlockCipher, CipherParameters cipherParameters, boolean forEncryption) throws InvalidCipherTextException {
bufferedBlockCipher.init(forEncryption, cipherParameters);
int inputOffset = 0;
int inputLength = input.length;
int maximumOutputLength = bufferedBlockCipher.getOutputSize(inputLength);
byte[] output = new byte[maximumOutputLength];
int outputOffset = 0;
int outputLength = 0;
int bytesProcessed;
bytesProcessed = bufferedBlockCipher.processBytes(
input, inputOffset, inputLength,
output, outputOffset
);
outputOffset += bytesProcessed;
outputLength += bytesProcessed;
bytesProcessed = bufferedBlockCipher.doFinal(output, outputOffset);
outputOffset += bytesProcessed;
outputLength += bytesProcessed;
if (outputLength == output.length) {
return output;
} else {
byte[] truncatedOutput = new byte[outputLength];
System.arraycopy(
output, 0,
truncatedOutput, 0,
outputLength
);
return truncatedOutput;
}
}
Edit: Whoops, I just read the article you linked to. It sounds like he is talking about even higher level abstractions than I thought (e.g., "send a confidential message"). I am afraid I don't quite understand what he is getting at.
Assuming that you write your application in Java I'd recommend that you don't use a specific provider, but that you develop your application on top of Sun's JCE (Java Cryptography Extension). Doing so can make you independent of any underlying providers, I.e., you can switch providers easily as long as you use ciphers that are widely implemented. It does give you a certain level of abstraction as you don't have to know all the details of the implementations and may protect you a little from using the wrong classes (e.g. such as using raw encryption without proper padding etc) Furthermore, Sun provides a decent amount of documentation and code samples.
I've actually found that this sample uses default 128 bit encryption instead of 256 bit. I've made a little change:
BlockCipher blockCipher = new AESEngine();
now becomes:
BlockCipher blockCipher = new RijndaelEngine(256);
and it works together with my client application C++ AES256 encryption
One example of a high(er)-level API in BouncyCastle would be the CMS (Cryptographic Message Syntax) package. This ships in a separate jar (bcmail) from the provider itself, and is written to the JCE (The C# version is written against the lightweight API however).
"Send a confidential message" is implemented, roughly speaking, by the CMSEnvelopedDataGenerator class, and all you really need to do is give it the message, choose an encryption algorithm (all details handled internally), and then specify one or more ways that a recipient will be able to read the message: this can be based on a public key/certificate, a shared secret, a password, or even a key agreement protocol. You can have more than one recipient on a message, and you can mix and match types of recipient.
You can use CMSSignedDataGenerator to similarly send a verifiable message. If you want to sign and encrypt, the CMS structures are nestable/composable (but order could be important). There's also CMSCompressedDataGenerator and recently added CMSAuthenticatedData.
You may use:
byte[] process(bool encrypt, byte[] input, byte[] key)
{
var cipher = CipherUtilities.GetCipher("Blowfish");
cipher.Init(false, new KeyParameter(key));
return cipher.DoFinal(input);
}
// Encrypt:
byte[] encrypted = process(true, clear, key);
// Decrypt:
byte[] decrypted = process(false, encrypted, key);
See: https://github.com/wernight/decrypt-toolbox/blob/master/dtDecrypt/Program.cs
JCE won't work for me because we want 256 bit strength and can't change the java configuration on the system to allow it. Too bad the Bouncy Castle doesn't have an API as high-level as JCE.
"Note however that bouncycastle consists of two libraries, the lightweight crypto library and the JCE provider interface library. The keysize restrictions are enforced by the JCE layer, but you don't need to use this layer. If you just use the lightweight crypto API directly you don't have any restrictions, no matter what policy files are or are not installed."
http://www.coderanch.com/t/420255/Security/AES-cryptoPerms-Unlimited-Cryptography
The book Beginning Cryptography with Java contains very helpful examples and explanations based on the bouncycastle library