I want to schedule a task on Hazelcast that runs at a fixed interval and updates the IMap with some data that I get after hitting a rest endpoint. Below is a sample code:
// Main class
IScheduledExecutorService service = hazelcast.getScheduledExecutorService("default");
service.scheduleAtFixedRate(TaskUtils.named("my-task", myTask), 30, 1);
// Task
#Singleton
public class MyTask implements Runnable, Serializable {
RestClient restClient;
IMap<String, JsonObject> map;
#Inject
MyTask() { // Inject hazelcast and restclient
map = hazelcastInstace.getMap("my-map");
this.restClient = restClient;
}
#Override
public void run() {
Collection<JSONObject> values = map.values(new MyCustomFilter());
for(JSONObject obj : values) {
// query endpoint based on id
map.submitToKey(key, response);
}
}
private static class MyCustomFilter implements Predicate<String, JSONObject> {
public boolean apply(Map.Entry<String, JSONObject> map) {
// logic to filter relevant keys
}
}
}
When I try to execute this on the cluster, I get:
java.io.NotSerializableException: com.hazelcast.map.impl.proxy.MapProxyImpl
Now I need the IMap to selectively query only some keys based on PredicateFilter and this needs to be a background scheduled job so stuck here on how to take this forward. Any help appreciated. TIA
Try making your task also implement HazelcastInstanceAware
When you submit your task, it is serialized, sent to the grid to run, deserialized when it is received, and the run() method is called.
If your task implements HazelcastInstanceAware, then between deserialization and run(), Hazelcast will call the method setHazelcastInstance(HazelcastInstance instance) to pass your code a reference to the particular Hazelcast instance it is running in. From there you can just do instance.getMap("my-map") and store the map reference in a transient field that the run() method can use.
Related
I have this flow that I am trying to test but nothing works as expected. The flow itself works well but testing seems a bit tricky.
This is my flow:
#Configuration
#RequiredArgsConstructor
public class FileInboundFlow {
private final ThreadPoolTaskExecutor threadPoolTaskExecutor;
private String filePath;
#Bean
public IntegrationFlow fileReaderFlow() {
return IntegrationFlows.from(Files.inboundAdapter(new File(this.filePath))
.filterFunction(...)
.preventDuplicates(false),
endpointConfigurer -> endpointConfigurer.poller(
Pollers.fixedDelay(500)
.taskExecutor(this.threadPoolTaskExecutor)
.maxMessagesPerPoll(15)))
.transform(new UnZipTransformer())
.enrichHeaders(this::headersEnricher)
.transform(Message.class, this::modifyMessagePayload)
.route(Map.class, this::channelsRouter)
.get();
}
private String channelsRouter(Map<String, File> payload) {
boolean isZip = payload.values()
.stream()
.anyMatch(file -> isZipFile(file));
return isZip ? ZIP_CHANNEL : XML_CHANNEL; // ZIP_CHANNEL and XML_CHANNEL are PublishSubscribeChannel
}
#Bean
public SubscribableChannel xmlChannel() {
var channel = new PublishSubscribeChannel(this.threadPoolTaskExecutor);
channel.setBeanName(XML_CHANNEL);
return channel;
}
#Bean
public SubscribableChannel zipChannel() {
var channel = new PublishSubscribeChannel(this.threadPoolTaskExecutor);
channel.setBeanName(ZIP_CHANNEL);
return channel;
}
//There is a #ServiceActivator on each channel
#ServiceActivator(inputChannel = XML_CHANNEL)
public void handleXml(Message<Map<String, File>> message) {
...
}
#ServiceActivator(inputChannel = ZIP_CHANNEL)
public void handleZip(Message<Map<String, File>> message) {
...
}
//Plus an #Transformer on the XML_CHANNEL
#Transformer(inputChannel = XML_CHANNEL, outputChannel = BUS_CHANNEL)
private List<BusData> xmlFileToIngestionMessagePayload(Map<String, File> xmlFilesByName) {
return xmlFilesByName.values()
.stream()
.map(...)
.collect(Collectors.toList());
}
}
I would like to test multiple cases, the first one is checking the message payload published on each channel after the end of fileReaderFlow.
So I defined this test classe:
#SpringBootTest
#SpringIntegrationTest
#ExtendWith(SpringExtension.class)
class FileInboundFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#TempDir
static Path localWorkDir;
#BeforeEach
void setUp() {
copyFileToTheFlowDir(); // here I copy a file to trigger the flow
}
#Test
void checkXmlChannelPayloadTest() throws InterruptedException {
Thread.sleep(1000); //waiting for the flow execution
PublishSubscribeChannel xmlChannel = this.getBean(XML_CHANNEL, PublishSubscribeChannel.class); // I extract the channel to listen to the message sent to it.
xmlChannel.subscribe(message -> {
assertThat(message.getPayload()).isInstanceOf(Map.class); // This is never executed
});
}
}
As expected that test does not work because the assertThat(message.getPayload()).isInstanceOf(Map.class); is never executed.
After reading the documentation I didn't find any hint to help me solved that issue. Any help would be appreciated! Thanks a lot
First of all that channel.setBeanName(XML_CHANNEL); does not effect the target bean. You do this on the bean creation phase and dependency injection container knows nothing about this setting: it just does not consult with it. If you really would like to dictate an XML_CHANNEL for bean name, you'd better look into the #Bean(name) attribute.
The problem in the test that you are missing the fact of async logic of the flow. That Files.inboundAdapter() works if fully different thread and emits messages outside of your test method. So, even if you could subscribe to the channel in time, before any message is emitted to it, that doesn't mean your test will work correctly: the assertThat() will be performed on a different thread. Therefore no real JUnit report for your test method context.
So, what I'd suggest to do is:
Have Files.inboundAdapter() stopped in the beginning of the test before any setup you'd like to do in the test. Or at least don't place files into that filePath, so the channel adapter doesn't emit messages.
Take the channel from the application context and if you wish subscribe or use a ChannelInterceptor.
Have an async barrier, e.g. CountDownLatch to pass to that subscriber.
Start the channel adapter or put file into the dir for scanning.
Wait for the async barrier before verifying some value or state.
I am new to Reactive programming and Spring WebFlux. I want to make my App 1 publish Server Sent event through Flux and my App 2 listen on it continuously.
I want Flux publish on-demand (e.g. when something happens). All the example I found is to use Flux.interval to periodically publish event, and there seems no way to append/modify the content in Flux once it is created.
How can I achieve my goal? Or I am totally wrong conceptually.
Publish "dynamically" using FluxProcessor and FluxSink
One of the techniques to supply data manually to the Flux is using FluxProcessor#sink method as in the following example
#SpringBootApplication
#RestController
public class DemoApplication {
final FluxProcessor processor;
final FluxSink sink;
final AtomicLong counter;
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
public DemoApplication() {
this.processor = DirectProcessor.create().serialize();
this.sink = processor.sink();
this.counter = new AtomicLong();
}
#GetMapping("/send")
public void test() {
sink.next("Hello World #" + counter.getAndIncrement());
}
#RequestMapping(produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent> sse() {
return processor.map(e -> ServerSentEvent.builder(e).build());
}
}
Here, I created DirectProcessor in order to support multiple subscribers, that will listen to the data stream. Also, I provided additional FluxProcessor#serialize which provide safe support for multiproducer (invocation from different threads without violation of Reactive Streams spec rules, especially rule 1.3). Finally, by calling "http://localhost:8080/send" we will see the message Hello World #1 (of course, only in case if you connected to the "http://localhost:8080" previously)
Update For Reactor 3.4
With Reactor 3.4 you have a new API called reactor.core.publisher.Sinks. Sinks API offers a fluent builder for manual data-sending which lets you specify things like the number of elements in the stream and backpressure behavior, number of supported subscribers, and replay capabilities:
#SpringBootApplication
#RestController
public class DemoApplication {
final Sinks.Many sink;
final AtomicLong counter;
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
public DemoApplication() {
this.sink = Sinks.many().multicast().onBackpressureBuffer();
this.counter = new AtomicLong();
}
#GetMapping("/send")
public void test() {
EmitResult result = sink.tryEmitNext("Hello World #" + counter.getAndIncrement());
if (result.isFailure()) {
// do something here, since emission failed
}
}
#RequestMapping(produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent> sse() {
return sink.asFlux().map(e -> ServerSentEvent.builder(e).build());
}
}
Note, message sending via Sinks API introduces a new concept of emission and its result. The reason for such API is the fact that the Reactor extends Reactive-Streams and has to follow the backpressure control. That said if you emit more signals than was requested, and the underlying implementation does not support buffering, your message will not be delivered. Therefore, the result of tryEmitNext returns the EmitResult which indicates if the message was sent or not.
Also, note, that by default Sinsk API gives a serialized version of Sink, which means you don't have to care about concurrency. However, if you know in advance that the emission of the message is serial, you may build a Sinks.unsafe() version which does not serialize given messages
Just another idea, using EmitterProcessor as a gateway to flux
import reactor.core.publisher.EmitterProcessor;
import reactor.core.publisher.Flux;
public class MyEmitterProcessor {
EmitterProcessor<String> emitterProcessor;
public static void main(String args[]) {
MyEmitterProcessor myEmitterProcessor = new MyEmitterProcessor();
Flux<String> publisher = myEmitterProcessor.getPublisher();
myEmitterProcessor.onNext("A");
myEmitterProcessor.onNext("B");
myEmitterProcessor.onNext("C");
myEmitterProcessor.complete();
publisher.subscribe(x -> System.out.println(x));
}
public Flux<String> getPublisher() {
emitterProcessor = EmitterProcessor.create();
return emitterProcessor.map(x -> "consume: " + x);
}
public void onNext(String nextString) {
emitterProcessor.onNext(nextString);
}
public void complete() {
emitterProcessor.onComplete();
}
}
More info, see here from Reactor doc. There is a recommendation from the document itself that "Most of the time, you should try to avoid using a Processor. They are harder to use correctly and prone to some corner cases." BUT I don't know which kind of corner case.
1. Is it possible to put non-POJO class instances as the value of a cache?
For example, I have a QueryThread class which is a subclass of java.lang.Thread and I am trying to put this instance in a cache. It looks like the put operation is failing because this cache is always empty.
Consider the following class:
public class QueryThread extends Thread {
private IgniteCache<?, ?> cache;
private String queryId;
private String query;
private long timeIntervalinMillis;
private volatile boolean running = false;
public QueryThread(IgniteCache<?, ?> dataCache, String queryId, String query, long timeIntervalinMillis) {
this.queryId = queryId;
this.cache = dataCache;
this.query = query;
this.timeIntervalinMillis = timeIntervalinMillis;
}
public void exec() throws Throwable {
SqlFieldsQuery qry = new SqlFieldsQuery(query, false);
while (running) {
List<List<?>> queryResult = cache.query(qry).getAll();
for (List<?> list : queryResult) {
System.out.println("result : "+list);
}
System.out.println("..... ");
Thread.sleep(timeIntervalinMillis);
}
}
}
This class is not a POJO. How do I store an instance of this class in the cache?
I tried implementing Serializable (didn't help).
I need to be able to do this:
queryCache.put(queryId, queryThread);
Next I tried broadcasting the class using the IgniteCallable interface. But my class takes multiple arguments in the constructor. I feel PeerClassLoading is easy if the class takes a no-arg constructor:
IgniteCompute compute = ignite.compute(ignite.cluster().forServers());
compute.broadcast(new IgniteCallable<MyServiceImpl>() {
#Override
public MyServiceImpl call() throws Exception {
MyServiceImpl myService = new MyServiceImpl();
return myService;
}
});
2. How do I do PeerClassLoading in the case of a class with multi-arg constructor?
It's restricted to put Thread instances to the cache, Thread instance cannot be serialized due to call to Native Methods. Thats why you always get empty value.
PeerClassLoading is a special distributed ClassLoader in Ignite for inter-node byte-code exchange. So, it's only about sharing classes between nodes. It doesn't make sense how many arguments in constructor class have.
But, on the other hand, object, that you created, will be serialised and sent to other nodes and for deserialisation it will need a default(non-arg) constructor.
I have a WCF service which has its Thread.CurrentPrincipal set in the ServiceConfiguration.ClaimsAuthorizationManager.
When I implement the service asynchronously like this:
public IAsyncResult BeginMethod1(AsyncCallback callback, object state)
{
// Audit log call (uses Thread.CurrentPrincipal)
var task = Task<int>.Factory.StartNew(this.WorkerFunction, state);
return task.ContinueWith(res => callback(task));
}
public string EndMethod1(IAsyncResult ar)
{
// Audit log result (uses Thread.CurrentPrincipal)
return ar.AsyncState as string;
}
private int WorkerFunction(object state)
{
// perform work
}
I find that the Thread.CurrentPrincipal is set to the correct ClaimsPrincipal in the Begin-method and also in the WorkerFunction, but in the End-method it's set to a GenericPrincipal.
I know I can enable ASP.NET compatibility for the service and use HttpContext.Current.User which has the correct principal in all methods, but I'd rather not do this.
Is there a way to force the Thread.CurrentPrincipal to the correct ClaimsPrincipal without turning on ASP.NET compatibility?
Starting with a summary of WCF extension points, you'll see the one that is expressly designed to solve your problem. It is called a CallContextInitializer. Take a look at this article which gives CallContextInitializer sample code.
If you make an ICallContextInitializer extension, you will be given control over both the BeginXXX thread context AND the EndXXX thread context. You are saying that the ClaimsAuthorizationManager has correctly established the user principal in your BeginXXX(...) method. In that case, you then make for yourself a custom ICallContextInitializer which either assigns or records the CurrentPrincipal, depending on whether it is handling your BeginXXX() or your EndXXX(). Something like:
public object BeforeInvoke(System.ServiceModel.InstanceContext instanceContext, System.ServiceModel.IClientChannel channel, System.ServiceModel.Channels.Message request){
object principal = null;
if (request.Properties.TryGetValue("userPrincipal", out principal))
{
//If we got here, it means we're about to call the EndXXX(...) method.
Thread.CurrentPrincipal = (IPrincipal)principal;
}
else
{
//If we got here, it means we're about to call the BeginXXX(...) method.
request.Properties["userPrincipal"] = Thread.CurrentPrincipal;
}
...
}
To clarify further, consider two cases. Suppose you implemented both an ICallContextInitializer and an IParameterInspector. Suppose that these hooks are expected to execute with a synchronous WCF service and with an async WCF service (which is your special case).
Below are the sequence of events and the explanation of what is happening:
Synchronous Case
ICallContextInitializer.BeforeInvoke();
IParemeterInspector.BeforeCall();
//...service executes...
IParameterInspector.AfterCall();
ICallContextInitializer.AfterInvoke();
Nothing surprising in the above code. But now look below at what happens with asynchronous service operations...
Asynchronous Case
ICallContextInitializer.BeforeInvoke(); //TryGetValue() fails, so this records the UserPrincipal.
IParameterInspector.BeforeCall();
//...Your BeginXXX() routine now executes...
ICallContextInitializer.AfterInvoke();
//...Now your Task async code executes (or finishes executing)...
ICallContextInitializercut.BeforeInvoke(); //TryGetValue succeeds, so this assigns the UserPrincipal.
//...Your EndXXX() routine now executes...
IParameterInspector.AfterCall();
ICallContextInitializer.AfterInvoke();
As you can see, the CallContextInitializer ensures you have opportunity to initialize values such as your CurrentPrincipal just before the EndXXX() routine runs. It therefore doesn't matter that the EndXXX() routine assuredly is executing on a different thread than did the BeginXXX() routine. And yes, the System.ServiceModel.Channels.Message object which is storing your user principal between Begin/End methods, is preserved and properly transmitted by WCF even though the thread changed.
Overall, this approach allows your EndXXX(IAsyncresult) to execute with the correct IPrincipal, without having to explicitly re-establish the CurrentPrincipal in the EndXXX() routine. And as with any WCF behavior, you can decide if this applies to individual operations, all operations on a contract, or all operations on an endpoint.
Not really the answer to my question, but an alternate approach of implementing the WCF service (in .NET 4.5) that does not exhibit the same issues with Thread.CurrentPrincipal.
public async Task<string> Method1()
{
// Audit log call (uses Thread.CurrentPrincipal)
try
{
return await Task.Factory.StartNew(() => this.WorkerFunction());
}
finally
{
// Audit log result (uses Thread.CurrentPrincipal)
}
}
private string WorkerFunction()
{
// perform work
return string.Empty;
}
The valid approach to this is to create an extension:
public class SLOperationContext : IExtension<OperationContext>
{
private readonly IDictionary<string, object> items;
private static ReaderWriterLockSlim _instanceLock = new ReaderWriterLockSlim();
private SLOperationContext()
{
items = new Dictionary<string, object>();
}
public IDictionary<string, object> Items
{
get { return items; }
}
public static SLOperationContext Current
{
get
{
SLOperationContext context = OperationContext.Current.Extensions.Find<SLOperationContext>();
if (context == null)
{
_instanceLock.EnterWriteLock();
context = new SLOperationContext();
OperationContext.Current.Extensions.Add(context);
_instanceLock.ExitWriteLock();
}
return context;
}
}
public void Attach(OperationContext owner) { }
public void Detach(OperationContext owner) { }
}
Now this extension is used as a container for objects that you want to persist between thread switching as OperationContext.Current will remain the same.
Now you can use this in BeginMethod1 to save current user:
SLOperationContext.Current.Items["Principal"] = OperationContext.Current.ClaimsPrincipal;
And then in EndMethod1 you can get the user by typing:
ClaimsPrincipal principal = SLOperationContext.Current.Items["Principal"];
EDIT (Another approach):
public IAsyncResult BeginMethod1(AsyncCallback callback, object state)
{
var task = Task.Factory.StartNew(this.WorkerFunction, state);
var ec = ExecutionContext.Capture();
return task.ContinueWith(res =>
ExecutionContext.Run(ec, (_) => callback(task), null));
}
While migrating a JBoss 5 application to JBoss AS 7 (7.1.1.FINAL) I have a problem with a new JMS message driven EJB. Within message processing, some master data fields have to be checked. To enhance performance, this master data shall be preloaded into a cache structure using a #Singleton #Startup EJB, which needs about 30 seconds to load the data.
My problem is that the queue message processing starts even if the cache has not been fully initialized, causing message validation errors.
I tried to define a dependency between the MDB and the startup EJB, but as far as I understood the #DependsOn annotation works only with #Singleton EJBs. So it's clear that my solution does not work ;-)
Startup bean code:
#Singleton
#Startup
public class StartupBean {
#PostConstruct
void atStartup() {
// TODO load master data cache (takes about 30 seconds)
}
#PreDestroy()
void atShutdown() {
// TODO free master data cache
}
}
Note: I stripped the real code from the example to make it easier to read :-)
Message driven bean code:
#MessageDriven(name="SampleMessagingBean", activationConfig = {
#ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
#ActivationConfigProperty(propertyName="destination", propertyValue="jms/SampleQueue"),
#ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge")
})
#DependsOn("StartupBean")
public class SampleMessagingBean implements MessageListener {
public void onMessage(Message message) {
// TODO validate message using master data cache
}
}
Question: How can I suspend message processing until the startup bean has finished loading the cache?
Any suggestions greatly appreciated :-)!
First i thought inject singleton EJB in mdb would be enough to delay message consumption
But no, sometimes it would start consuming the message before #PostConstruct of Singleton-ejb completed. So added a method invocation also and it started working
This worked on glassfish, but i dont see a reason why it shouldnt work on jboss
Singleton-Ejb:
#Singleton
#Startup
public class SingletonBean {
private Logger logger = Logger.getLogger(getClass().getName());
private boolean init = false;
public boolean isInit() {
return init;
}
#PostConstruct
public void init() {
logger.error("singleton init start");
//Do something that takes time here
init = true;
logger.error("singleton init end ");
}
}
and mdb:
#MessageDriven(...)
public class SomeMdb implements MessageListener {
private Logger logger = Logger.getLogger(getClass().getName());
#EJB
SingletonBean sb;
#PostConstruct
public void init() {
logger.error("mdb init start");
if (!sb.isInit()) {
logger.error("never happens");
}
logger.error("mdb init complete");
}
public void onMessage(Message message) {
logger.error("onMessage start");
}
}
Now it always waits for SingletonBean to complete init before mdb completes init (as seen in log)
19:51:51,980 [ad-pool-1; w: 3] ERROR SomeMdb - mdb init start
19:51:52,122 [ad-pool-4848(4)] ERROR SingletonBean - singleton init start
19:51:56,316 [ad-pool-4848(4)] ERROR SingletonBean - singleton init end
19:51:56,317 [ad-pool-1; w: 3] ERROR SomeMdb - mdb init complete
19:51:56,317 [ad-pool-1; w: 3] ERROR SomeMdb - onMessage start