I am trying to read kafka messages from KafkaSpout and set tuple values from json that are parsed from that message. Actually, I am creating an additional Bolt that parses a tuple field called "value" with json string from KafkaSpout. Is it possible to set these values in Spout?
class ScanConfigKafkaSpout(kafkaUrl: String, kafkaGroup: String, kafkaTopic: String) : KafkaSpout<String, String>(
KafkaSpoutConfig
.builder(kafkaUrl, kafkaTopic)
.setProp(KEY_KAFKA_GROUP, "grp1")
.setProcessingGuarantee(KafkaSpoutConfig.ProcessingGuarantee.AT_MOST_ONCE)
.build()
), ComponentId {
override fun open(conf: MutableMap<String, Any>?, context: TopologyContext?, collector: SpoutOutputCollector?) {
try {
logger.debug("<${id()}> Opening ScanConfigKafkaSpout with ${conf.toString()}")
super.open(conf, context, collector)
logger.debug("<${id()}> ScanConfigKafkaSpout opened")
} catch (t: Throwable) {
logger.error("<${id()}> Error during opening CrawlScanConfigKafkaSpout", t)
}
}
override fun id(): String = SCAN_CONFIG_KAFKA_SPOUT
companion object {
private val logger = LoggerFactory.getLogger(ScanConfigKafkaSpout::class.java)
}
}
You probably need to implement the method declareOutputFields(OutputFieldsDeclarer declarer from IComponent.
It is used by Storm to serialize your attribute values and tuple configurations.
As stated here in the section Data Model , it says:
Every node in a topology must declare the output fields for the tuples it emits.
There is also a java example given for that method.
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("double", "triple"));
}
Related
I'm trying to build a class where certain values are Observable but also Serializable.
This obviously works and the serialization works, but it's very boilerplate-heavy having to add a setter for every single field and manually having to call change(...) inside each setter:
interface Observable {
fun change(message: String) {
println("changing $message")
}
}
#Serializable
class BlahVO : Observable {
var value2: String = ""
set(value) {
field = value
change("value2")
}
fun toJson(): String {
return Json.encodeToString(serializer(), this)
}
}
println(BlahVO().apply { value2 = "test2" })
correctly outputs
changing value2
{"value2":"test2"}
I've tried introducing Delegates:
interface Observable {
fun change(message: String) {
println("changing $message")
}
#Suppress("ClassName")
class default<T>(defaultValue: T) {
private var value: T = defaultValue
operator fun getValue(observable: Observable, property: KProperty<*>): T {
return value
}
operator fun setValue(observable: Observable, property: KProperty<*>, value: T) {
this.value = value
observable.change(property.name)
}
}
}
#Serializable
class BlahVO : Observable {
var value1: String by Observable.default("value1")
fun toJson(): String {
return Json.encodeToString(serializer(), this)
}
}
println(BlahVO().apply { value1 = "test1" }) correctly triggers change detection, but it doesn't serialize:
changing value1
{}
If I go from Observable to ReadWriteProperty,
interface Observable {
fun change(message: String) {
println("changing $message")
}
fun <T> look(defaultValue: T): ReadWriteProperty<Observable, T> {
return OP(defaultValue, this)
}
class OP<T>(defaultValue: T, val observable: Observable) : ObservableProperty<T>(defaultValue) {
override fun setValue(thisRef: Any?, property: KProperty<*>, value: T) {
super.setValue(thisRef, property, value)
observable.change("blah!")
}
}
}
#Serializable
class BlahVO : Observable {
var value3: String by this.look("value3")
fun toJson(): String {
return Json.encodeToString(serializer(), this)
}
}
the result is the same:
changing blah!
{}
Similarly for Delegates.vetoable
var value4: String by Delegates.vetoable("value4", {
property: KProperty<*>, oldstring: String, newString: String ->
this.change(property.name)
true
})
outputs:
changing value4
{}
Delegates just doesn't seem to work with Kotlin Serialization
What other options are there to observe a property's changes without breaking its serialization that will also work on other platforms (KotlinJS, KotlinJVM, Android, ...)?
Serialization and Deserialization of Kotlin Delegates is not supported by kotlinx.serialization as of now.
There is an open issue #1578 on GitHub regarding this feature.
According to the issue you can create an intermediate data-transfer object, which gets serialized instead of the original object. Also you could write a custom serializer to support the serialization of Kotlin Delegates, which seems to be even more boilerplate, then writing custom getters and setters, as proposed in the question.
Data Transfer Object
By mapping your original object to a simple data transfer object without delegates, you can utilize the default serialization mechanisms.
This also has the nice side effect to cleanse your data model classes from framework specific annotations, such as #Serializable.
class DataModel {
var observedProperty: String by Delegates.observable("initial") { property, before, after ->
println("""Hey, I changed "${property.name}" from "$before" to "$after"!""")
}
fun toJson(): String {
return Json.encodeToString(serializer(), this.toDto())
}
}
fun DataModel.toDto() = DataTransferObject(observedProperty)
#Serializable
class DataTransferObject(val observedProperty: String)
fun main() {
val data = DataModel()
println(data.toJson())
data.observedProperty = "changed"
println(data.toJson())
}
This yields the following result:
{"observedProperty":"initial"}
Hey, I changed "observedProperty" from "initial" to "changed"!
{"observedProperty":"changed"}
Custom data type
If changing the data type is an option, you could write a wrapping class which gets (de)serialized transparently. Something along the lines of the following might work.
#Serializable
class ClassWithMonitoredString(val monitoredProperty: MonitoredString) {
fun toJson(): String {
return Json.encodeToString(serializer(), this)
}
}
fun main() {
val monitoredString = obs("obsDefault") { before, after ->
println("""I changed from "$before" to "$after"!""")
}
val data = ClassWithMonitoredString(monitoredString)
println(data.toJson())
data.monitoredProperty.value = "obsChanged"
println(data.toJson())
}
Which yields the following result:
{"monitoredProperty":"obsDefault"}
I changed from "obsDefault" to "obsChanged"!
{"monitoredProperty":"obsChanged"}
You however lose information about which property changed, as you don't have easy access to the field name. Also you have to change your data structures, as mentioned above and might not be desirable or even possible. In addition, this work only for Strings for now, even though one might make it more generic though.
Also, this requires a lot of boilerplate to start with. On the call site however, you just have to wrap the actual value in an call to obs.
I used the following boilerplate to get it to work.
typealias OnChange = (before: String, after: String) -> Unit
#Serializable(with = MonitoredStringSerializer::class)
class MonitoredString(initialValue: String, var onChange: OnChange?) {
var value: String = initialValue
set(value) {
onChange?.invoke(field, value)
field = value
}
}
fun obs(value: String, onChange: OnChange? = null) = MonitoredString(value, onChange)
object MonitoredStringSerializer : KSerializer<MonitoredString> {
override val descriptor: SerialDescriptor = PrimitiveSerialDescriptor("MonitoredString", PrimitiveKind.STRING)
override fun serialize(encoder: Encoder, value: MonitoredString) {
encoder.encodeString(value.value)
}
override fun deserialize(decoder: Decoder): MonitoredString {
return MonitoredString(decoder.decodeString(), null)
}
}
I'm quite new to Kotlin and I'd like to see if using high-order functions can help in my case.
My use-case is that I need to call the methods of an IInterface derived class to send events to one or more components. And I'd like to make this generic, and I want to check if a high-order funtion can help. A sample of code will help to understand (well, I hope so!).
private val eventListeners = mutableListOf<IEventInterface>() // List filled somewhere else!
private fun sendConnectionEvent(dummyString: String) {
val deadListeners = mutableListOf<IEventInterface>()
eventListeners.forEach {
try {
it.onConnectionEvent(dummyString)
} catch (e: DeadObjectException) {
Log.d(TAG, "Removing listener - Exception ${e.message}")
deadListeners.add(it)
}
}
deadListeners.forEach { it ->
eventListeners.remove(it)
}
}
private fun sendWonderfulEvent(dummyString: String, dummyInt: Int) {
val deadListeners = mutableListOf<IEventInterface>()
eventListeners.forEach {
try {
it.onWonderfulEvent(dummyString, dummyInt)
} catch (e: DeadObjectException) {
Log.d(TAG, "Removing listener - Exception ${e.message}")
deadListeners.add(it)
}
}
deadListeners.forEach { it ->
eventListeners.remove(it)
}
}
I added 2 similar methods (I will have many more in the real use case) and I think (I hope!) that something could be done but I can't make high-order function works in this case because:
I want to call the same method on several instances, and not 'just' a basic function
To make things even worse, the methods I need to call don't have the same prototype (that would have been too easy!).
Hope this is clear enough.
Thanks for your help!
VR
Here is how it can be done
fun onEvent(body: (IEventInterface) -> Unit) {
val deadListeners = mutableListOf<IEventInterface>()
eventListeners.forEach {
try {
body(it)
} catch (ex: DeadObjectException) {
Log.d(TAG, "Removing listener - Exception ${e.message}")
deadListeners.add(it)
}
}
deadListeners.forEach { it ->
eventListeners.remove(it)
}
}
Supposing an interface like this:
interface IEventInterface {
fun onConnectionEvent(dummyString: String)
fun onWonderfulEvent(dummyString: String, dummyInt: Int)
}
Define an generic type that implements your defined interface ( <T : IEventInterface>)
Define an mutable list of this type to receive your implementation (MutableList<T>.removeIfThrows)
Expect an extension function for you type that will do your specific validation (and custom parameters if you want)
Using an apply and returning the instance you can run your code like a pipeline
Executing the custom validation when you want
private fun <T : IEventInterface> MutableList<T>.removeIfThrows(validation: T.() -> Unit, customLogMessage: String? = null): MutableList<T> {
return apply {
removeIf {
it.runCatching {
validation()
}.onFailure { error ->
print(customLogMessage ?: "Removing listener - Exception ${error.message}")
}.isFailure
}
}
}
Define your specific implementation passing just the function with custom validation as an parameter
private fun <T : IEventInterface> MutableList<T>.sendConnectionEvent(dummyString: String) = removeIfThrows({
onConnectionEvent(dummyString)
})
private fun <T : IEventInterface> MutableList<T>.sendWonderfulEvent(dummyString: String, dummyInt: Int) = removeIfThrows({
onWonderfulEvent(dummyString, dummyInt)
})
Now you can run your code like an pipeline modifying your original object like this
private fun nowYouCanDoSomethingLikeThis() {
eventListeners
.sendConnectionEvent("some dummy string")
.sendWonderfulEvent("some another dummy string", 123)
}
I am implementing a declarative client in Micronaut that looks like this:
#Get("/dostuff{?requestObject*}")
fun getStuff(requestObject: MyRequestObject): String
My MyRequestObject contains an enum that is represented by some string:
data class MyRequestObject(val myEnum: MyEnum)
enum class MyEnum(val stringRep: String) {
AREASONABLENAME("someSillyString");
}
When I now send a request via the client the value from requestObject generates the following query /?myEnum=AREASONABLENAME. What I actually need is /?myEnum=someSillyString.
I tried the following things without any success:
add JsonValue function to MyEnum:
#JsonValue fun getJsonValue() = stringRep - of course did not help
implement a TypeConverter for MyEnum
#Singleton
class MyEnumTypeConverter : TypeConverter<MyEnum, String> {
override fun convert(`object`: MyEnum?, targetType: Class<String>?, context: ConversionContext?): Optional<String> {
return Optional.ofNullable(`object`?.stringRep)
}
}
Is there a way to achieve the desired behaviour?
You can override the toString method in the Enum so that when the converter tries to convert it to a string you can control the result of the operation:
enum class MyEnum(val stringRep: String) {
AREASONABLENAME("someSillyString");
override fun toString(): String {
return stringRep
}
}
I am refactoring and adding to the API communication of an app. I'd like to get to this usage for my "json data objects". Instantiate with either the properties directly or from a json string.
userFromParams = User("user#example.com", "otherproperty")
userFromString = User.fromJson(someJsonString)!!
// userIWantFromString = User(someJsonString)
Getting userFromParams to serialize to JSON was not a problem. Just adding a toJson() function takes care of that.
data class User(email: String, other_property: String) {
fun toJson(): String {
return Moshi.Builder().build()
.adapter(User::class.java)
.toJson(this)
}
companion object {
fun fromJson(json: String): User? {
val moshi = Moshi.Builder().add(KotlinJsonAdapterFactory()).build()
return moshi.adapter(User::class.java).fromJson(json)
}
}
}
It is "fromJson" that I would like to get rid of ...because... I want to and I can't figure out how. The above class works (give or take wether to allow an optional object to be returned or not and so on) but it just bugs me that I get stuck trying to get to this nice clean overloaded initialization.
It does not strictly have to be a data class either, but it does seem appropriate here.
You can't really do that in any performant way. Any constructor invocation will instantiate a new object, but since Moshi handles object creation internally, you'll have two instances...
If you really REALLY want it though, you can try something like:
class User {
val email: String
val other_property: String
constructor(email: String, other_property: String) {
this.email = email
this.other_property = other_property
}
constructor(json: String) {
val delegate = Moshi.Builder().build().adapter(User::class.java).fromJson(json)
this.email = delegate.email
this.other_property = delegate.other_property
}
fun toJson(): String {
return Moshi.Builder()
.add(KotlinJsonAdapterFactory())
.build()
.adapter(User::class.java)
.toJson(this)
}
}
By default, ThreeTenABP.LocalDateTime is converted to
{"date":{"day":10,"month":4,"year":2018},"time":{"hour":3,"minute":34,"nano":115000000,"second":18}}
I can write an adapter to support ISO date string 2018-04-10T03:45:26.009
class LocalDateTimeAdapter {
#ToJson
fun toJson(value: LocalDateTime): String {
return FORMATTER.format(value)
}
#FromJson
fun fromJson(value: String): LocalDateTime {
return FORMATTER.parse(value, LocalDateTime.FROM)
}
companion object {
private val FORMATTER = DateTimeFormatter.ISO_LOCAL_DATE_TIME
}
}
How can I write an adapter which can support both format (fromJson)
{"date":{"day":10,"month":4,"year":2018},"time":{"hour":3,"minute":34,"nano":115000000,"second":18}}
2018-04-10T03:45:26.009
Beside identifying which the format is used in fromJson, I am curious how Moshi internally perform toJson/fromJson for LocalDateTime
You’ll need to use JsonReader.peek() to determine the format of the incoming JSON, and then take action accordingly.
First install an adapter that converts LocalDateTime to a string. That adapter should use a qualifier annotation.
#Retention(RetentionPolicy.RUNTIME)
#JsonQualifier
#interface DateString {
}
Next create the string adapter. It should be straightforward, and might delegate to Moshi’s built-in Rfc3339DateJsonAdapter.
public final class LocalDateAsStringAdapter {
#ToJson String toJson(#DateString LocalDateTime localDateTime) {
...
}
#FromJson #DateString LocalDateTime fromJson(String string) {
...
}
}
Finally create an adapter that delegates either to Moshi’s built in adapter (that one will use {...}) or to your string adapter. This one prefers the string format, but you can do what you like.
public final class MultipleFormatsDateAdapter {
#ToJson void toJson(JsonWriter writer, LocalDateTime value,
#DateString JsonAdapter<LocalDateTime> stringAdapter) throws IOException {
stringAdapter.toJson(writer, value);
}
#FromJson LocalDateTime fromJson(JsonReader reader, #DateString JsonAdapter<LocalDateTime> stringAdapter,
JsonAdapter<LocalDateTime> defaultAdapter) throws IOException {
if (reader.peek() == JsonReader.Token.STRING) {
return stringAdapter.fromJson(reader);
} else {
return defaultAdapter.fromJson(reader);
}
}
}
This works because Moshi lets you declare multiple JsonAdapter arguments to the #ToJson and #FromJson methods, and these arguments may be annotated.
It also relies on the way this feature works if the types are the same. Here we’re making a JsonAdapter<LocalDateTime> by delegating to another JsonAdapter<LocalDateTime>. When the types are the same Moshi uses its nextAdapter() feature for composition.