Apache Ignite Caching and PeerClassLoading - ignite

1. Is it possible to put non-POJO class instances as the value of a cache?
For example, I have a QueryThread class which is a subclass of java.lang.Thread and I am trying to put this instance in a cache. It looks like the put operation is failing because this cache is always empty.
Consider the following class:
public class QueryThread extends Thread {
private IgniteCache<?, ?> cache;
private String queryId;
private String query;
private long timeIntervalinMillis;
private volatile boolean running = false;
public QueryThread(IgniteCache<?, ?> dataCache, String queryId, String query, long timeIntervalinMillis) {
this.queryId = queryId;
this.cache = dataCache;
this.query = query;
this.timeIntervalinMillis = timeIntervalinMillis;
}
public void exec() throws Throwable {
SqlFieldsQuery qry = new SqlFieldsQuery(query, false);
while (running) {
List<List<?>> queryResult = cache.query(qry).getAll();
for (List<?> list : queryResult) {
System.out.println("result : "+list);
}
System.out.println("..... ");
Thread.sleep(timeIntervalinMillis);
}
}
}
This class is not a POJO. How do I store an instance of this class in the cache?
I tried implementing Serializable (didn't help).
I need to be able to do this:
queryCache.put(queryId, queryThread);
Next I tried broadcasting the class using the IgniteCallable interface. But my class takes multiple arguments in the constructor. I feel PeerClassLoading is easy if the class takes a no-arg constructor:
IgniteCompute compute = ignite.compute(ignite.cluster().forServers());
compute.broadcast(new IgniteCallable<MyServiceImpl>() {
#Override
public MyServiceImpl call() throws Exception {
MyServiceImpl myService = new MyServiceImpl();
return myService;
}
});
2. How do I do PeerClassLoading in the case of a class with multi-arg constructor?

It's restricted to put Thread instances to the cache, Thread instance cannot be serialized due to call to Native Methods. Thats why you always get empty value.
PeerClassLoading is a special distributed ClassLoader in Ignite for inter-node byte-code exchange. So, it's only about sharing classes between nodes. It doesn't make sense how many arguments in constructor class have.
But, on the other hand, object, that you created, will be serialised and sent to other nodes and for deserialisation it will need a default(non-arg) constructor.

Related

Hangfire - DisableConcurrentExecution - Prevent concurrent execution if same value passed in method parameter

Hangfire DisableConcurrentExecution attribute not working as expected.
I have one method and that can be called with different Id. I want to prevent concurrent execution of method if same Id is passed.
string jobName= $"{Id} - Entry Job";
_recurringJobManager.AddOrUpdate<EntryJob>(jobName, j => j.RunAsync(Id, Null), "0 2 * * *");
My EntryJob interface having RunAsync method.
public class EntryJob: IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
public async Task RunAsync(int Id, SomeObj obj)
{
//Some coe
}
}
And interface look like this
[DisableConcurrentExecution(3600)] <-- Tried here
public interface IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
Task RunAsync(int Id, SomeObj obj);
}
Now I want to prevent RunAsync method to call multiple times if Id is same. I have tried to put DisableConcurrentExecution on top of the RunAsync method at both location inside interface declaration and also from where Interface is implemented.
But it seems like not working for me. Is there any way to prevent concurrency based on Id?
The existing implementation of DisableConcurrentExecution does not support this. It will prevent concurrent executions of the method with any args. It would be fairly simple to add support in. Note below is untested pseudo-code:
public class DisableConcurrentExecutionWithArgAttribute : JobFilterAttribute, IServerFilter
{
private readonly int _timeoutInSeconds;
private readonly int _argPos;
// add additional param to pass in which method arg you want to use for
// deduping jobs
public DisableConcurrentExecutionAttribute(int timeoutInSeconds, int argPos)
{
if (timeoutInSeconds < 0) throw new ArgumentException("Timeout argument value should be greater that zero.");
_timeoutInSeconds = timeoutInSeconds;
_argPos = argPos;
}
public void OnPerforming(PerformingContext filterContext)
{
var resource = GetResource(filterContext.BackgroundJob.Job);
var timeout = TimeSpan.FromSeconds(_timeoutInSeconds);
var distributedLock = filterContext.Connection.AcquireDistributedLock(resource, timeout);
filterContext.Items["DistributedLock"] = distributedLock;
}
public void OnPerformed(PerformedContext filterContext)
{
if (!filterContext.Items.ContainsKey("DistributedLock"))
{
throw new InvalidOperationException("Can not release a distributed lock: it was not acquired.");
}
var distributedLock = (IDisposable)filterContext.Items["DistributedLock"];
distributedLock.Dispose();
}
private static string GetResource(Job job)
{
// adjust locked resource to include the argument to make it unique
// for a given ID
return $"{job.Type.ToGenericTypeString()}.{job.Method.Name}.{job.Args[_argPos].ToString()}";
}
}

How to schedule a task on Hazelcast that queries on the IMap?

I want to schedule a task on Hazelcast that runs at a fixed interval and updates the IMap with some data that I get after hitting a rest endpoint. Below is a sample code:
// Main class
IScheduledExecutorService service = hazelcast.getScheduledExecutorService("default");
service.scheduleAtFixedRate(TaskUtils.named("my-task", myTask), 30, 1);
// Task
#Singleton
public class MyTask implements Runnable, Serializable {
RestClient restClient;
IMap<String, JsonObject> map;
#Inject
MyTask() { // Inject hazelcast and restclient
map = hazelcastInstace.getMap("my-map");
this.restClient = restClient;
}
#Override
public void run() {
Collection<JSONObject> values = map.values(new MyCustomFilter());
for(JSONObject obj : values) {
// query endpoint based on id
map.submitToKey(key, response);
}
}
private static class MyCustomFilter implements Predicate<String, JSONObject> {
public boolean apply(Map.Entry<String, JSONObject> map) {
// logic to filter relevant keys
}
}
}
When I try to execute this on the cluster, I get:
java.io.NotSerializableException: com.hazelcast.map.impl.proxy.MapProxyImpl
Now I need the IMap to selectively query only some keys based on PredicateFilter and this needs to be a background scheduled job so stuck here on how to take this forward. Any help appreciated. TIA
Try making your task also implement HazelcastInstanceAware
When you submit your task, it is serialized, sent to the grid to run, deserialized when it is received, and the run() method is called.
If your task implements HazelcastInstanceAware, then between deserialization and run(), Hazelcast will call the method setHazelcastInstance(HazelcastInstance instance) to pass your code a reference to the particular Hazelcast instance it is running in. From there you can just do instance.getMap("my-map") and store the map reference in a transient field that the run() method can use.

NSubstitute: Received Calls asserts wrongly

I've created this test:
[TestFixture]
public class UsersTests
{
private Core.Kernel coreKernel;
private Core.Configuration.ICoreConfiguration coreConfiguration;
[SetUp]
public void SetUp()
{
this.coreConfiguration = NSubstitute.Substitute.For<Core.Configuration.ICoreConfiguration>();
this.coreKernel = NSubstitute.Substitute.For<Core.Kernel>(this.coreConfiguration);
this.coreKernel.Initialize();
}
[Test]
public void AddUserTest()
{
Core.Communication.Entities.UserIdentity receivedUserIdentity = new Core.Communication.Entities.UserIdentity("user1", "passwd1");
((Core.Communication.ICoreService)this.coreKernel).AddUserIdentity(receivedUserIdentity);
this.coreKernel.Received(100).AddUser(Arg.Is<Core.Identity.UserIdentity>(u => u.UserId.Equals(receivedUserIdentity.UserId)));
}
}
where Core.Kernel is:
public partial class Kernel : Core.IKernel
{
public Kernel(Configuration.ICoreConfiguration configuration)
: this(configuration, null, Enumerable.Empty<Type>())
{
}
public Kernel(Configuration.ICoreConfiguration configuration, Communication.ICoreService service, IEnumerable<Type> producerTypes)
{
if (configuration == null)
throw new ArgumentException("configuration object must be provided", "configuration");
if (producerTypes.Any(t => !t.IsAssignableFrom(typeof(Core.Extensibility.AbstractProducerPlugin))))
throw new ArgumentException("All types must inherit from AbstractProducerPlugin", "plugins");
this.state = KernelState.initializing;
this.configuration = configuration;
this.service = service ?? this;
this.producerTypes = producerTypes;
this.backends = new Dictionary<Core.Identity.DomainIdentity, Backend.Infrastructure.IBackend>();
}
internal virtual void AddUser(Core.Identity.UserIdentity userIdentity) {...}
}
Nevertheless, this.coreKernel.Received(100).AddUser(... is not called 100 times, only one. What am I doing wrong?
I mean, I'm not trying to make 100 calls to AddUser. I'm checking AddUser should be called 100 times. So, assertion should fail.
EDIT
Guess this code (Core.IKernel.AddUserIdentity(...) implementation):
public class Core.Kernel {
public override void Core.IKernel.AddUserIdentity(UserIdentity userIdentity) {
this.AddUser(userIdentity); <<----- AddUser(...) is called
}
}
I think the problem is related with:
Core.Kernel implements Core.IKernel. Core.IKernel has AddUserIdentity(...) method.
I'm mocking Core.Kernel instead of mocking a Core.IKernel.
According to Core.IKernel.AddUserIdentity(...) method implementation AddUser should ne reached.
AddUser is an internal virtual method of Core.Kernel. It's not an implementation of any method interface.
I want to assert AddUser is called once when AddUserIdentity is reached.
Other questions about mocking:
For<T> where T is a concrete class -> virtual methods are replaced? no virtual methods are executed?
ForPartsOf<T> where T is a concrete class -> Which parts of this class are mocked (virtual methods, overrided interface methods)?
It is only called once because you are only calling AddUser once. The Received assertion checks how many times it has been called, it doesn't tell NSubstitue to call your method 100 times, you need to do that manually :)
NSubstitute also gives you the option of asserting a specific number of calls were received by passing an integer to Received(). This will throw if the substitute does not receive exactly that many matching calls
http://nsubstitute.github.io/help/received-calls/

Hazelcast 3.6.1 "There is no suitable de-serializer for type" exception

I am using Hazelcast 3.6.1 to read from a Map. The object class stored in the map is called Schedule.
I have configured a custom serializer on the client side like this.
ClientConfig config = new ClientConfig();
SerializationConfig sc = config.getSerializationConfig();
sc.addSerializerConfig(add(new ScheduleSerializer(), Schedule.class));
...
private SerializerConfig add(Serializer serializer, Class<? extends Serializable> clazz) {
SerializerConfig sc = new SerializerConfig();
sc.setImplementation(serializer).setTypeClass(clazz);
return sc;
}
The map is created like this
private final IMap<String, Schedule> map = client.getMap("schedule");
If I get from the map using schedule id as key, the map returns the correct value e.g.
return map.get("zx81");
If I try to use an SQL predicate e.g.
return new ArrayList<>(map.values(new SqlPredicate("statusActive")));
then I get the following error
Exception in thread "main" com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable de-serializer for type 2. This exception is likely to be caused by differences in the serialization configuration between members or between clients and members.
The custom serializer is using Kryo to serialize (based on this blog http://blog.hazelcast.com/comparing-serialization-methods/)
public class ScheduleSerializer extends CommonSerializer<Schedule> {
#Override
public int getTypeId() {
return 2;
}
#Override
protected Class<Schedule> getClassToSerialize() {
return Schedule.class;
}
}
The CommonSerializer is defined as
public abstract class CommonSerializer<T> implements StreamSerializer<T> {
protected abstract Class<T> getClassToSerialize();
#Override
public void write(ObjectDataOutput objectDataOutput, T object) {
Output output = new Output((OutputStream) objectDataOutput);
Kryo kryo = KryoInstances.get();
kryo.writeObject(output, object);
output.flush(); // do not close!
KryoInstances.release(kryo);
}
#Override
public T read(ObjectDataInput objectDataInput) {
Input input = new Input((InputStream) objectDataInput);
Kryo kryo = KryoInstances.get();
T result = kryo.readObject(input, getClassToSerialize());
input.close();
KryoInstances.release(kryo);
return result;
}
#Override
public void destroy() {
// empty
}
}
Do I need to do any configuration on the server side? I thought that the client config would be enough.
I am using Hazelcast client 3.6.1 and have one node/member running.
Queries require the nodes to know about the classes as the bytestream has to be deserialized to access the attributes and query them. This means that when you want to query on objects you have to deploy the model classes (and serializers) on the server side as well.
Whereas when you use key-based access we do not need to look into the values (neither into the keys as we compare the byte-arrays of the key) and just send the result. That way neither model classes nor serializers have to be available on the Hazelcast nodes.
I hope that makes sense.

How to force MOXy to use the setter on a Collection property that is lazily initialized?

Given a bean like this:
public class MyBean {
private List<Something> things;
private List<Something> internalGetThings() {
if (things == null) {
things = new ArrayList<Something>();
}
return things;
}
public Iterable<Something> getThings() {
return <an immutable copy of internalGetThings()>;
}
public void setThings(List<Something> someThings) {
things.clear();
for (Something aThing : someThings) {
addThing(aThing);
}
}
public void addThing(Something aThing) {
things.add(aThing);
// Do some special stuff to aThing
}
}
Using external mapping file, when I map like this:
<xml-element java-attribute="things" name="thing" type="com.myco.Something" container-type="java.util.ArrayList" />
It seems that each individual Something is being added to the MyBean by calling getThings().add(). That's a problem because getThings() returns an immutable copy of the list, which is lazily initialized. How can I configure mapping (I'm using an external mapping file, not annotations) so that MOXy uses setThings() or addThing() instead?
Why Does JAXB/MOXy Check the Get Method for Collection First?
JAXB (JSR-222) implementations give you a chance to have your property be the List interface and still leverage the underlying List implementation that you choose to use. To accomplish this a JAXB implementation will call the get method to see if the List implementation has been initialized. It it has the List will be populated using the add method.
public List<String> getThings() {
if(null == things) {
things = new ArrayList<String>();
}
return things;
}
public List<String> getThings() {
if(null == things) {
things = new LinkedList<String>();
}
return things;
}
If you don't pre-initialize the List property then MOXy/JAXB will build an instance of the List (default is ArrayList) and set it on the object using the set method.
private List<Something> things; // Don't Initialize
public List<String> getThings() {
return things;
}
public void setThings(List<String> things) {
this.things = things;
}
Given the reason in #Blaise's answer, it doesn't seem possible to have MOXy (or any JAXB implementation in general?) populate a lazily-initialized collection via a setter method on the collection. However, a combination of xml-accessor-type="FIELD" (or #XmlAccessorType if using annotations) and defining a JAXB unmarshal event callback will get the job done. In my afterUnmarshal() implementation I do the special work on Something instances that is done in addSomething().
private void afterUnmarshal(Unmarshaller, Object parent) {
for (Something aThing : getSomethings()) {
// Do special stuff on aThing
}
}
Using FIELD access type gets JAXB/MOXy to directly inject the collection into the field, bypassing the getter. Then the call back cleans things up properly.