I am trying send data types like int, float, bolean etc and also Java Object.
I dont see any examples or implementations for this kind of use cases. Those examples I saw they have only string.
In short:
with Rabbitmq you can send a buffer, you can create the buffer as you prefer, for example:
byte[] messageBodyBytes = ByteBuffer.allocate(4).putInt(yourint).array();
channel.basicPublish(exchangeName, routingKey, null, messageBodyBytes);
if you want to send a java class, you can serialize it using JSON format.
Long:
You sholud read the amqp protocol specification here : https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf
You need to serialize/deserialize the data you want to send. Perhaps using JSON or XML or some other format that's convenient to your app
You can serialze any Java Object that implements Serializable interface.
Related
I have a Kafka consumer configured with schema polling from the topic, what I would like to do, is create another Avro schema, on top of the current one, and hydrate data using it, basically I don't need 50% of the information and need to write some logic to change a couple of fields. Thats just an example
val consumer: KafkaConsumer<String, GenericRecord>(props) = createConsumer()
while (true) {
consumer.poll(Duration.ofSeconds(10).forEach {it ->
println(it.value())
}
}
The event returned from stream is pretty complex, so I've modelled a smaller CustomObj as a .avsc file and compiled it to java. And when trying to run the code with the CustomObj, Error deserializing key/value for partition all I want to do is consume an event, and then deserialize it into a much smaller object with just selected fields.
return KafkaConsumer<String, CustomObj>(props)
This didn't work, not sure how can I deserialize it using CustomObj from the GenericRecord? Let me just add that I don't have any access to the stream or its config I can just consume from it.
In Avro, your reader schema needs to be compatible with the writer schema. By giving the smaller object, you're providing a different reader schema
It's not possible to directly deserialize to a subset of the input data, so you must parse the larger object and map it to the smaller one (which isn't what deserialization does)
I have a Mongo collection annotated with #Document and I want the possibility to also get that Java object from a String (JSON) as we're getting these classes pushed into a queue as String.
Is there a method in Spring-Data-Mongo which converts from JSON to the actual Document object?
#Autowired
MongoTemplate mongoTemplate;
and then
mongoTemplate.getConverter().read(MatchMongo.class, (DBObject) JSON.parse(json));
Thanks to freakman, your answer helped a lot
You can try com.mongodb.util.JSON.parse() method. It returns object so you probably have to do the casting + it may be it need "class" field inside json string.
HI Looking for APIs to write parquest with Pojos that I have.
I was able to generate avro schema using reflection and then create parquet schema using AvroSchemaConverter.
Also i am not able to find a way to convert Pojos to GenericRecords (avro) else I could have been able to use AvroParquetWriter to write out the Pojos into parquet files.
Any suggestions ?
If you want to go through avro you have two options:
1) Let avro generate your pojos (see the tutorial here). The generated pojos extend SpecificRecord which can then be used with AvroParquetWriter.
2) Write the conversion from your pojo to GenericRecord yourself. You can do this either manually or a more generic solution would be to use reflection. However, I encountered difficulties with this approach when I tried to read the data. Based on the supplied schema avro found the pojo in the classpath and tried to instantiate a SpecificRecord instead of GenericRecord. Because of this reason I went with option 1.
Parquet also supports now writing pojo directly. Here is the pull request on parquet github page. However, I think this is not part of an official release yet. In another words, I did not find this code in maven.
DISCLAIMER: The following code was written when I was in a hurry. It is not efficient and future versions of parquet will surely fix this more directly. That being said, this is a lightweight inefficient approach to what you need. The strategy is POJO -> AVRO -> PARQUET
POJO -> AVRO: Declare a schema via reflection. Declare writers and readers based on the schema. At the time of conversion write the object to byte stream and read it back as avro.
AVRO -> Parquet: use the AvroParquetWriter included in the parquet-me project.
private static final Schema avroSchema = ReflectData.AllowNull.get().getSchema(YOURCLASS.class);
private static final ReflectDatumWriter<YOURCLASS> reflectDatumWriter = new ReflectDatumWriter<>(avroSchema);
private static final GenericDatumReader<Object> genericRecordReader = new GenericDatumReader<>(avroSchema);
public GenericRecord toAvroGenericRecord() throws IOException {
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
reflectDatumWriter.write(this, EncoderFactory.get().directBinaryEncoder(bytes, null));
return (GenericRecord) genericRecordReader.read(null, DecoderFactory.get().binaryDecoder(bytes.toByteArray(), null));
}
One more thing: it seems the parquet writers are currently very strict about null fields. Make sure none of your fields are null before attempting to write to parquet
I wasn't able to find an existing solution, so I implemented it myself. Here is the link to the implementation: https://gist.github.com/alexeygrigorev/eab72e40c6051e0163a6693054906d66
In short, it does the following:
uses reflection to get Avro schema from the pojo
using the schema and reflection it converts pojos to GenericRecord objects
reflection is applied recursively if the pojo contains other pojos or list of pojos
com / twitter / finagle / memcached / Client extends BaseClient[ChannelBuffer]
so the set is defined as
set(key: String, flags: Int, expiry: Time, value: ChannelBuffer): Future[Unit]
My question is how can I convert my java Object to a netty ChannelBuffer or I have some other way to set object other than String?
Thanks
Finally, I used thrift to do the serialization. It's just my case, because I have heavily used thrift.
The disadvantage of thrift serialization: you had to define the struct of the object in the IDL.
The advantage is: simple
TSerializer serializer = new TSerializer();
byte[] bytes = serializer.serialize(obj);
ChannelBuffer buffer = ChannelBufferUtils.bytesToChannelBuffer(bytes);
It's up to you!
You can use whatever serialization library you want, there is plenty of alternatives.
memcached only consider it as a array of bytes.
I know returning types in a wcv service is allowed, and it will convert the object to json. But what if I don't want to return a random type, but return a string with formatted json? I can construct json my self but it can a) be messy, and b) not auto encode html values.
How do I do build a custom formatted json string? Off the top of my had, can I return a dictionary with key, value pairs? Will the value be encoded so you can transmitted without running the risk of malformed json?
Have a look at JSON.Net. I've used it in the past for serializing/deserializing to/from Json. It also (according to the web page) has support for converting Json to/from XML, so it seems reasonable that there would be some functions in there to build arbitrary Json strings in a way that is less error-prone than doing it yourself.
You can specify a return type of object and then use an anonymous type to return an arbitrary object. If you want to return an arbitrary collection, you can return IEnumerable, and use that to return a collection of anonymous types.
as far as I can understand, you want a webservice that returns a string that can be parsed using json (like JSON.parse(yourReturnedString)... As ckramer answered, JSON.NET can help to format your whatever dictionary into json but you should know dictionary is "json-serialised" as key:'your key', value:'your value§that can be also an object that will be serialized', so if you are using JSON.NET, you should also once it has been deserialezed, remove all the "key": and ,"value" JSON.NET returned.
so good so far you should definetely declare your webmethod as a method that returns a JSON format.
hope you found a solution before this answer...