In java class with annotation:
public final class Info {
private Info() {
}
public static class InfoAction {
public static final String OPEN = "open";
public static final String VIEW_ALL = "view_all";
#Retention(RetentionPolicy.SOURCE)
#StringDef({OPEN, VIEW_ALL})
public #interface Action {
}
public String mAction;
public InfoAction(#Action String action) {
this.mAction = action;
}
}
IDE convert to kotlin:
class Info private constructor() {
class InfoAction(#param:Action var infoAction: String) {
#kotlin.annotation.Retention(AnnotationRetention.SOURCE)
#StringDef(OPEN, VIEW_ALL)
annotation class Action
companion object {
const val OPEN = "open"
const val VIEW_ALL = "view_all"
}
}
}
it has #param:Action, but replace with #Action it works as well.
what is this #param here for, and can the #Action be used?
#param is for constructor parameter
detail:
https://kotlinlang.org/docs/annotations.html#annotation-use-site-targets
I've recently been learning about how to use the Kafka Streams client and one thing that I've been struggling with is how to switch from the default state store (RocksDB) to a custom state store using something like Redis. The Confluent documentation makes it clear you have to implement the StateStore interface for your custom store and you must provide an implementation of StoreBuilder for creating instances of that store.
Here's what I have so far for my custom store. I've also added a simple write method to append a new entry into a specified stream via the Redis XADD command.
public class MyCustomStore<K,V> implements StateStore, MyWriteableCustomStore<K,V> {
private String name;
private volatile boolean open = false;
private boolean loggingEnabled = false;
public MyCustomStore(String name, boolean loggingEnabled) {
this.name = name;
this.loggingEnabled = loggingEnabled;
}
#Override
public String name() {
return this.name;
}
#Override
public void init(ProcessorContext context, StateStore root) {
if (root != null) {
// register the store
context.register(root, (key, value) -> {
write(key.toString(), value.toString());
});
}
this.open = true;
}
#Override
public void flush() {
// TODO Auto-generated method stub
}
#Override
public void close() {
// TODO Auto-generated method stub
}
#Override
public boolean persistent() {
// TODO Auto-generated method stub
return true;
}
#Override
public boolean isOpen() {
// TODO Auto-generated method stub
return false;
}
#Override
public void write(String key, String value) {
try(Jedis jedis = new Jedis("localhost", 6379)) {
Map<String, String> hash = new HashMap<>();
hash.put(key, value);
jedis.xadd("MyStream", StreamEntryID.NEW_ENTRY, hash);
}
}
}
public class MyCustomStoreBuilder implements StoreBuilder<MyCustomStore<String,String>> {
private boolean cached = true;
private String name;
private Map<String,String> logConfig=new HashMap<>();
private boolean loggingEnabled;
public MyCustomStoreBuilder(String name, boolean loggingEnabled){
this.name = name;
this.loggingEnabled = loggingEnabled;
}
#Override
public StoreBuilder<MyCustomStore<String,String>> withCachingEnabled() {
this.cached = true;
return this;
}
#Override
public StoreBuilder<MyCustomStore<String,String>> withCachingDisabled() {
this.cached = false;
return null;
}
#Override
public StoreBuilder<MyCustomStore<String,String>> withLoggingEnabled(Map<String, String> config) {
loggingEnabled=true;
return this;
}
#Override
public StoreBuilder<MyCustomStore<String,String>> withLoggingDisabled() {
this.loggingEnabled = false;
return this;
}
#Override
public MyCustomStore<String,String> build() {
return new MyCustomStore<String,String>(this.name, this.loggingEnabled);
}
#Override
public Map<String, String> logConfig() {
return logConfig;
}
#Override
public boolean loggingEnabled() {
return loggingEnabled;
}
#Override
public String name() {
return name;
}
}
And here's what my setup and topology look like.
#Bean
public KafkaStreams kafkaStreams(KafkaProperties kafkaProperties) {
final Properties props = new Properties();
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaProperties.getBootstrapServers());
props.put(StreamsConfig.APPLICATION_ID_CONFIG, appName);
props.put(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Long().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Double().getClass());
props.put(StreamsConfig.STATE_DIR_CONFIG, "data");
props.put(StreamsConfig.APPLICATION_SERVER_CONFIG, appServerConfig);
props.put(JsonDeserializer.VALUE_DEFAULT_TYPE, JsonNode.class);
props.put(DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, LogAndContinueExceptionHandler.class);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
final String storeName = "the-custome-store";
Topology topology = new Topology();
// Create CustomStoreSupplier for store name the-custom-store
MyCustomStoreBuilder customStoreBuilder = new MyCustomStoreBuilder(storeName, false);
topology.addSource("input","inputTopic");
topology.addProcessor("redis-processor", () -> new RedisProcessor(storeName), "input");
topology.addStateStore(customStoreBuilder, "redis-processor");
KafkaStreams kafkaStreams = new KafkaStreams(topology, props);
kafkaStreams.start();
return kafkaStreams;
}
public class MyCustomStoreType<K,V> implements QueryableStoreType<MyReadableCustomStore<String,String>> {
#Override
public boolean accepts(StateStore stateStore) {
return stateStore instanceof MyCustomStore;
}
#Override
public MyReadableCustomStore<String,String> create(final StateStoreProvider storeProvider, final String storeName) {
return new MyCustomStoreTypeWrapper<>(storeProvider, storeName, this);
}
}
public class MyCustomStoreTypeWrapper<K,V> implements MyReadableCustomStore<K,V> {
private final QueryableStoreType<MyReadableCustomStore<String, String>> customStoreType;
private final String storeName;
private final StateStoreProvider provider;
public MyCustomStoreTypeWrapper(final StateStoreProvider provider,
final String storeName,
final QueryableStoreType<MyReadableCustomStore<String, String>> customStoreType) {
this.provider = provider;
this.storeName = storeName;
this.customStoreType = customStoreType;
}
#Override
public String read(String key) {
try (Jedis jedis = new Jedis("localhost", 6379)) {
StreamEntryID start = new StreamEntryID(0, 0);
StreamEntryID end = null; // null -> until the last item in the stream
int count = 2;
List<StreamEntry> list = jedis.xrange("MyStream", start, end, count);
if (list != null) {
// Get the most recently added item, which is also the last item
StreamEntry streamData = list.get(list.size() - 1);
return streamData.toString();
} else {
System.out.println("No new data in the stream");
}
return "";
}
}
}
// This throws the InvalidStateStoreException when I try to get access to the custom store
MyReadableCustomStore<String,String> store = streams.store("the-custome-store", new MyCustomStoreType<String,String>());
String value = store.read("testKey");
So, my question is how do I actually get the state store data to persist into Redis now? I feel like I'm missing something in the state store initialization or with the StateRestoreCallback. Any help or clarification with this would be greatly appreciated.
It looks to me that you have the store wired up to the topology correctly. But you don't have any processors using the store.
It could look something like this:
final String storeName = "the-custome-store";
MyCustomStoreBuilder customStoreBuilder = new MyCustomStoreBuilder(storeName, false);
Topology topology = new Topology()
topology.addSource("input", "input-topic");
// makes the processor a child of the source node
// the source node forwards its records to the child processor node
topology.addProcessor("redis-processor", () -> new RedisProcessor(storeName), "input");
// add the store and specify the processor(s) that access the store
topology.addStateStore(storeBuilder, "redis-processor");
class RedisProcessor implements Processor<byte[], byte[]> {
final String storeName;
MyCustomStore<byte[],byte[]> stateStore;
public RedisProcessor(String storeName) {
this.storeName = storeName;
}
#Override
public void init(ProcessorContext context) {
stateStore = (MyCustomeStore<byte[], byte[]>) context.getStateStore(storeName);
}
#Override
public void process(byte[] key, byte[] value) {
stateStore.write(key, value);
}
#Override
public void close() {
}
}
HTH, and let me know how it works out for you.
Update to answer from comments:
I think you need to update MyCustomStore.isOpen() to return the open variable.
Right now it's hardcoded to return false
Override
public boolean isOpen() {
// TODO Auto-generated method stub
return false;
}
How to clean autowired object after each test execution. In my example TestClassSettings object does not get clean property and it uses the previous test class value. Here is the example:
TestClassSettings.java
#Component
public class TestClassSettings{
private String titleUnitCode = "99";
private String fte = "1";
private String testIndicator = "";
public String getTitleUnitCode() {
return titleUnitCode;
}
public void setTitleUnitCode(String titleUnitCode) {
this.titleUnitCode = titleUnitCode;
}
public String getFte() {
return fte;
}
public void setFte(String fte) {
this.fte = fte;
}
public String getTestIndicator() {
return testIndicator;
}
public void setTestIndicator(String testIndicator) {
this.testIndicator = testIndicator;
}
}
testClassSettings instance is not getting clean in between test cases.
TestLeaveHourCal_bweh6.java
#TestMethodOrder(OrderAnnotation.class)
#SpringJUnitWebConfig(locations = { "classpath:service.xml"})
#TestInstance(Lifecycle.PER_CLASS)
public class TestLeaveHourCal_bweh6 {
#Autowired
private ApproveTimesheetService approveTimesheetService;
#Autowired
private ComparePayUpdates comparePayUpdates;
#Autowired
public TestClassSettings testClassSettings; /* variable access type needs public */;
#Autowired
#RegisterExtension
protected CreateTimesheetBeforeTestExecutionCallback beforeTestExecutionCallback; /* can not be private */
#BeforeAll
public void setup() throws Exception {
/* START SETTINGS */
testClassSettings.setTestIndicator("6");
testClassSettings.setTitleUnitCode("99");
testClassSettings.setFte("0.5");
/* END SETTINGS */
}
#Test
#Order(1)
#Tag("setBaselinePayUpdates")
public void setBaselinePayUpdates() throws Exception {
}
Added #DirtiesContext(classMode = ClassMode.AFTER_CLASS) to fix the issue
is there a way to map a DTO using MatStruct which have a few final data members as well and cannot have a default constructor , like :
public class TestDto {
private final String testName;
private int id;
private String testCase;
public TestDto(String testName) {
this.testName = testName;
}
public String getTestName() {
return testName;
}
public int getId() {
return id;
}
public String getTestCase() {
return testCase;
}
public void setId(int id) {
this.id = id;
}
public void setTestCase(String testCase) {
this.testCase = testCase;
}
}
please suggest how could this DTO be mapped using MapStruct.
You can use #ObjectFactory that would construct an instance of your DTO.
For example:
#Mapper
public interface MyMapper {
#ObjectFactory
default TestDto create() {
return new TestDto("My Test Name");
}
//the rest of the mappings
}
You can also enhance the #ObjectFactory to accept the source parameter, that you can use to construct the TestDto. You can even use a #Context as an Object Factory.
NB: You don't have to put the #ObjectFactory method in the same Mapper, or even a MapStruct #Mapper. You can put it in any class (or make it static) and then #Mapper(uses = MyFactory.class)
I'm continuing on with a logging behavior using the WebSocketBehavior. It currently logs the correct data to a console, but also throws a terrible serialization error. It is because I am providing the WicketApplication itself as a constructor argument for the behavior. I've tried passing it my session object and using that to get the WebApplication, but it consistently returns null. The broadcaster object requires the application in order to function properly. My question is how can I provide the WebApplication to the behavior while avoiding the nasty serialization error?? Here is my behavior class:
public class LogWebSocketBehavior extends WebSocketBehavior implements Serializable {
private static final long serialVersionUID = 1L;
private Console console;
private Handler logHandler;
private Model<LogRecord> model = new Model<>();
private WebApplication application;
public LogWebSocketBehavior(Console console, WebApplication application) {
super();
configureLogger();
this.console = console;
this.application = application;
}
private void configureLogger() {
Enumeration<String> list = LogManager.getLogManager().getLoggerNames();
list.hasMoreElements();
Logger l = Logger.getLogger(AppUtils.loggerName);
l.addHandler(getLoggerHandler());
}
#Override
protected synchronized void onPush(WebSocketRequestHandler handler, IWebSocketPushMessage message) {
LogRecord r = model.getObject();
sendRecordToConsole(handler, r);
}
private Handler getLoggerHandler() {
return new LogHandler() {
private static final long serialVersionUID = 1L;
#Override
public void publish(LogRecord record) {
model.setObject(record);
sendToAllConnectedClients("data");
}
};
}
private synchronized void sendToAllConnectedClients(String message) {
IWebSocketConnectionRegistry registry = new SimpleWebSocketConnectionRegistry();
WebSocketPushBroadcaster b = new WebSocketPushBroadcaster(registry);
b.broadcastAll(application, new Message());
}
private void sendRecordToConsole(WebSocketRequestHandler handler, LogRecord r) {
Level level = r.getLevel();
if (level.equals(Level.INFO)) {
console.info(handler, new SimpleFormatter().formatMessage(r));
} else {
console.error(handler, new SimpleFormatter().formatMessage(r));
}
}
class Message implements IWebSocketPushMessage {
public Message() {
}
}
}
Here is the panel that is being used to display the messages:
public class FooterPanel extends Panel {
private static final long serialVersionUID = 1L;
private Form form;
private Console console;
public FooterPanel(String id) {
super(id);
}
#Override
public void onInitialize() {
super.onInitialize();
form = new Form("form");
form.add(console = getConsole("feedback_console"));
console.setOutputMarkupId(true);
form.setOutputMarkupId(true);
add(form);
add(getLoggingBehavior());
}
private Console getConsole(String id) {
return new Console(id) {
private static final long serialVersionUID = 1L;
};
}
private WebSocketBehavior getLoggingBehavior() {
return new LogWebSocketBehavior(console, this.getWebApplication());
}
}
I updated my behavior as follows:
public class LogWebSocketBehavior extends WebSocketBehavior implements Serializable {
private static final long serialVersionUID = 1L;
private Console console;
private Handler logHandler;
private Model<LogRecord> model = new Model<>();
public LogWebSocketBehavior(Console console) {
super();
configureLogger();
this.console = console;
}
private void configureLogger() {
Enumeration<String> list = LogManager.getLogManager().getLoggerNames();
list.hasMoreElements();
Logger l = Logger.getLogger(AppUtils.loggerName);
l.addHandler(getLoggerHandler());
}
#Override
protected synchronized void onPush(WebSocketRequestHandler handler, IWebSocketPushMessage message) {
LogRecord r = model.getObject();
sendRecordToConsole(handler, r);
}
private Handler getLoggerHandler() {
return new LogHandler() {
private static final long serialVersionUID = 1L;
#Override
public void publish(LogRecord record) {
model.setObject(record);
sendToAllConnectedClients("data");
}
};
}
private synchronized void sendToAllConnectedClients(String message) {
WebApplication application = WebApplication.get();
IWebSocketConnectionRegistry registry = new SimpleWebSocketConnectionRegistry();
WebSocketPushBroadcaster b = new WebSocketPushBroadcaster(registry);
b.broadcastAll(application, new Message());
}
private void sendRecordToConsole(WebSocketRequestHandler handler, LogRecord r) {
Level level = r.getLevel();
String message = AppUtils.consoleDateTimeFormat.format(LocalDateTime.now()) + " - " + AppUtils.LogFormatter.formatMessage(r);
if (level.equals(Level.INFO)) {
console.info(handler, message);
} else {
console.error(handler, message);
}
}
class Message implements IWebSocketPushMessage {
public Message() {
}
}
}
And I'm back to the original issues I started with, which is the following error:
ERROR - ErrorLogger - Job (report.DB5E002E046235586592E7E984338DEE3 : 653 threw an exception.
org.quartz.SchedulerException:
Job threw an unhandled exception. [See nested exception: org.apache.wicket.WicketRuntimeException: There is no application attached to current thread DefaultQuartzScheduler_Worker-1]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: org.apache.wicket.WicketRuntimeException: There is no application attached to current thread DefaultQuartzScheduler_Worker-1
at org.apache.wicket.Application.get(Application.java:236)
at org.apache.wicket.protocol.http.WebApplication.get(WebApplication.java:160)
at eb.wicket.behaviors.LogWebSocketBehavior.sendToAllConnectedClients(LogWebSocketBehavior.java:77)
at eb.wicket.behaviors.LogWebSocketBehavior.access$100(LogWebSocketBehavior.java:29)
at eb.wicket.behaviors.LogWebSocketBehavior$1.publish(LogWebSocketBehavior.java:70)
Finally working as desired.. Here's the behavior class:
public class LogWebSocketBehavior extends WebSocketBehavior implements Serializable {
private static final long serialVersionUID = 1L;
private Console console;
private Model<LogRecord> model = new Model<>();
public LogWebSocketBehavior(Console console) {
super();
configureLogger();
this.console = console;
}
private void configureLogger() {
Enumeration<String> list = LogManager.getLogManager().getLoggerNames();
list.hasMoreElements();
Logger l = Logger.getLogger(AppUtils.loggerName);
l.addHandler(getLoggerHandler());
}
#Override
protected synchronized void onPush(WebSocketRequestHandler handler, IWebSocketPushMessage message) {
LogRecord r = model.getObject();
sendRecordToConsole(handler, r);
}
private Handler getLoggerHandler() {
return new LogHandler() {
private static final long serialVersionUID = 1L;
#Override
public void publish(LogRecord record) {
model.setObject(record);
sendToAllConnectedClients("data");
}
};
}
private synchronized void sendToAllConnectedClients(String message) {
IWebSocketConnectionRegistry registry = new SimpleWebSocketConnectionRegistry();
WebSocketPushBroadcaster b = new WebSocketPushBroadcaster(registry);
b.broadcastAll(Application.get("eb.wicket.MyWicketFilter"), new Message());
}
private void sendRecordToConsole(WebSocketRequestHandler handler, LogRecord r) {
Level level = r.getLevel();
String message = AppUtils.consoleDateTimeFormat.format(LocalDateTime.now()) + " - " + AppUtils.LogFormatter.formatMessage(r);
if (level.equals(Level.INFO)) {
console.info(handler, message);
} else {
console.error(handler, message);
}
}
class Message implements IWebSocketPushMessage {
public Message() {
}
}
}
Instead of keeping a reference to the Application just look it up when needed: Application.get().
After updating your question we can see:
Caused by: org.apache.wicket.WicketRuntimeException:
There is no application attached to current thread DefaultQuartzScheduler_Worker-1
This explains it - this is a thread started by Quartz, it is not a http thread.
The only way to overcome this is to use Application.get(String). The value should be the application name (Application#getName()) that is specified as a value for <filter-name> in your web.xml.
This way you can get the Application instance, but there is no way to do the same for Session and/or RequestCycle in case you need them too.