I have an InputStream (serialized bytes) containing marshalled Parcels with Parcelables inside them. I need to unmarshall and inflate the Parcel back up again so I can retrieve the Parcelable. How can I do this, when I don't know the length of the Parcelable? Is there a way to do this automatically just continuously reading from the bytestream?
Related
I have a Kafka consumer configured with schema polling from the topic, what I would like to do, is create another Avro schema, on top of the current one, and hydrate data using it, basically I don't need 50% of the information and need to write some logic to change a couple of fields. Thats just an example
val consumer: KafkaConsumer<String, GenericRecord>(props) = createConsumer()
while (true) {
consumer.poll(Duration.ofSeconds(10).forEach {it ->
println(it.value())
}
}
The event returned from stream is pretty complex, so I've modelled a smaller CustomObj as a .avsc file and compiled it to java. And when trying to run the code with the CustomObj, Error deserializing key/value for partition all I want to do is consume an event, and then deserialize it into a much smaller object with just selected fields.
return KafkaConsumer<String, CustomObj>(props)
This didn't work, not sure how can I deserialize it using CustomObj from the GenericRecord? Let me just add that I don't have any access to the stream or its config I can just consume from it.
In Avro, your reader schema needs to be compatible with the writer schema. By giving the smaller object, you're providing a different reader schema
It's not possible to directly deserialize to a subset of the input data, so you must parse the larger object and map it to the smaller one (which isn't what deserialization does)
I have an interface RequestProcessorInterface. There are different scenarios for processing a json request e.g. async vs synchronous request. I am trying to use template method pattern where the steps are like validate, preProcess, saveInDatabase, postProcess, etc
ProcessingContext processingContext = validate(storeContentRequestContainer);
processingContext = preProcess(storeContentRequestContainer, processingContext);
saveInDatabase(storeContentRequestContainer, processingContext);
return postProcess(storeContentRequestContainer, processingContext);
My ProcessingContext class has these attributes:
Map<String, String> errors;
String contentId;
String correlationId;
String presignedUrl;
String objectKey;
String bucketLocation;
DbEntity dbEntity; // Entity for database storage
When the json request is parsed, I assign values to 'processingContext' object. To keep the system flexible and not worry about what step might need the parsed information, I am encapsulating the extracted information in the context object.
Also I am passing the context object to every step so that in future, every step has this information readily available. I was going in the direction of most of the steps to be able to read the context and update some attribute and return the modified context , so the subsequent steps have access to attributes populated earlier.
I have a feeling that accepting context object (which is mutable) and modifying it and returning it is not a good idea. This context object is going to be in the method scope of a singleton class (spring boot). It will not be something lingering on forever and that should make it simpler.
How do I achieve this flexibility of multiple steps to be able to augment / update information in a context object? Will it make sense to make this context object immutable ?
I have an API PATCH endpoint that receives a Delta<T> and I need to serialize it, put it on a queue, and deserialize it in another process.
Using JsonConvert.SerializeObject returns {}. The incoming Delta<T> looks like it's correctly formed.
Any idea why Delta<T> is serializing to an empty JSON object?
UPDATE
Using System.Web.OData.Delta<T> gives me {}.
Using System.Web.Http.OData.Delta<T> gives me a full serialization of T, including unchanged properties.
Using System.Web.OData.Delta<T> gives me {}.
Using System.Web.Http.OData.Delta<T> gives me a full serialization of T, including unchanged properties.
I have some serialization in google protobuf in a series of files, but wonder if there is a shortcut way of concatenating these smaller files into one larger protobuf without worrying about reading each and every protobuf and then grouping these objects and outputting.
Is there a cheap way to join files together? I.e. do I have serialize each individual file?
You can combine protocol buffers messages by simple concatenation. It appears that you want the result to form an array, so you'll need to serialize each individual file as an array itself:
message MyItem {
...
}
message MyCollection {
repeated MyItem items = 1;
}
Now if you serialize each file as a MyCollection and then concatenate them (just put the raw binary data together), the resulting file can be read as a one large collection itself.
In addition to jpas answer, it might be relevant to say that the data does not need to be in the exact same container, when being serialized, for it being compatible on deserialisation.
Consider the following messages:
message FileData{
required uint32 versionNumber = 1;
repeated Data initialData = 2;
}
message MoreData{
repeated Data data = 2;
}
It is possible to serialize those different messages into one single data container and deserialize it as one single FileData message, as long as the FileData is serialized before zero or more MoreData and both, the FileData and MoreData have the same index for the repeated field.
I am using BufferManager in my WCF service. I created my own class to wrap around the BufferManager which implements IDisposable. Right now my Dispose method looks like this:
public void Dispose()
{
this.bufferManager.Clear();
}
My question: does this accomplish the same thing as calling ReturnBuffer on all of the buffers that have been taken by the manager?
Just for some background: I am using the BufferManager in the following method:
public byte[] ReadAllBufferedBytes(string filePath)
{
using (var fileStream =
new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
byte[] buffer = this.bufferManager.TakeBuffer((int)fileStream.Length);
fileStream.Read(buffer, 0, buffer.Length);
return buffer;
}
}
The reason I am doing this is because I kept getting OutOfMemory exceptions which would tear down the service.
The host server has 3 GB of memory. The service is in InstanceContextMode.Single mode, so images are processed one at a time. The images are received as byte arrays - the biggest might be 100MB, but are typically much smaller - converted, and then returned as a byte array. A lot ends up on the Large Object Heap, and image sizes vary quite a bit.
I am wondering if the issue is heap fragmentation.
As each page of a document gets converted, it is appended to a temp file on disk. After the conversion, I read the entire converted file from disk into a byte array and return it to the client.
The standard File.ReadAllBytes method creates a new byte array when it reads from the file, which inevitably ends up on the LOH due to the image sizes I'm working with (I assume this is what happens). I created the ReadAllBufferedBytes method to do the same thing, but to buffer the byte array and let the BufferManager return the buffer when it is disposed.
Another question is: do I even need to do all this?
The BufferManager is normally used in scenarios where you must prevent GC pressure - where there are a lot of small byte[] assignments, like when receiving or sending data on a very low level (e.g. sockets). The emphasis here is on a lot, otherwise the GC should be able to handle the memory allocations just fine.
To prevent the loading of the entire converted document file into memory you should use the FileStream directly (without reading it's entire content into memory - a byte[]) in combination with the streamed (response) TransferMode, if possible.