I have implemented a kind of caching bean to cache data objects as an EJB Singleton. I wonder if this is the correct way in EJB:
#Singleton
public class MyCache {
int DEFAULT_CACHE_SIZE = 30;
int DEFAULT_EXPIRES_TIME = 60000;
long expiresTime = 0;
long lastReset = 0;
Cache cache = null;
....
#PostConstruct
void init() {
resetCache();
}
public void resetCache() {
cache = new Cache(DEFAULT_CACHE_SIZE);
lastReset = System.currentTimeMillis();
}
public void put(String key, Object value) {
cache.put(key, value);
}
public Object get(String key) {
// test if cache is expired
if (expiresTime > 0) {
Long now = System.currentTimeMillis();
if ((now - lastReset) > expiresTime) {
logger.finest("...... Cache expired!");
resetCache();
}
}
return cache.get(key);
}
class Cache extends LinkedHashMap<String, Object> implements Serializable {
private static final long serialVersionUID = 1L;
private final int capacity;
public Cache(int capacity) {
super(capacity + 1, 1.1f, true);
this.capacity = capacity;
}
protected boolean removeEldestEntry(Entry<String, Object> eldest) {
return size() > capacity;
}
}
}
My question is: is this the right way to implement an application wide caching mechanism?
I have the impression that the contents of the cache are unexpectedly changing. Could this happen? For example, if the EJB is passivated?
I am running in a Payara41 Server.
Or must I use:
cache = Collections.synchronizedMap(new Cache(DEFAULT_CACHE_SIZE));
instead of:
cache = new Cache(DEFAULT_CACHE_SIZE);
First of all, as concurrency management is not specified for your bean, it falls down to default "container".
From EJB 3.1 spec:
When designing a Singleton session bean, the developer must decide
whether the bean will use container managed or bean managed
concurrency. Typically Singleton beans will be specified to have
container managed concurrency demarcation. This is the default if no
concurrency management type is specified.
Then, container concurrency management needs method-level specifications of lock type. As soon as those are absent, the default "Write" applies:
By default, if a concurrency locking attribute annotation is not
specified for a method of a Singleton bean with container managed
concurrency demarcation, the value of the concurrency locking
attribute for the method is defined to be Write.
The above means that access to your bean methods must be synchronized, probably even more than you actually need. You can set "Read" locking type for read-only methods (get), to allow concurrent read access.
The solution with container managed lock has one drawback,
consider if you have a put operation with WRITE lock it means the whole "cache" is blocked so no get or put is possible in parallel, no matter whether the key is different.
If your cache implementation is a concurrent Map you can use it without locking.
If you have more requirements for the cache I would use Infinispan which provide a better performance. The cache here can be local or distributed in a cluster.
After reading this blog from Adam Bien I now improved my code in the following way:
I added the annotation ConcurrencyManagement
I changed my LinkedHashMap into a ConcurrentHashMap.
Example:
#Singleton
#ConcurrencyManagement(ConcurrencyManagementType.BEAN) // added concurrency management
public class MyCache {
int DEFAULT_CACHE_SIZE = 30;
int DEFAULT_EXPIRES_TIME = 60000;
long expiresTime = 0;
long lastReset = 0;
Cache cache = null;
....
#PostConstruct
void init() {
resetCache();
}
public void resetCache() {
cache = new Cache(DEFAULT_CACHE_SIZE);
lastReset = System.currentTimeMillis();
}
public void put(String key, Object value) {
cache.put(key, value);
}
public Object get(String key) {
// test if cache is expired
if (expiresTime > 0) {
Long now = System.currentTimeMillis();
if ((now - lastReset) > expiresTime) {
logger.finest("...... Cache expired!");
resetCache();
}
}
return cache.get(key);
}
// changed from LinkedHashMap to ConcurrentHashMap
class Cache extends ConcurrentHashMap<String, Object> implements Serializable {
private static final long serialVersionUID = 1L;
private final int capacity;
public Cache(int capacity) {
super(capacity + 1, 1.1f);
this.capacity = capacity;
}
protected boolean removeEldestEntry(Entry<String, Object> eldest) {
return size() > capacity;
}
}
}
I think this is now the more correct implementation.
Related
Hangfire DisableConcurrentExecution attribute not working as expected.
I have one method and that can be called with different Id. I want to prevent concurrent execution of method if same Id is passed.
string jobName= $"{Id} - Entry Job";
_recurringJobManager.AddOrUpdate<EntryJob>(jobName, j => j.RunAsync(Id, Null), "0 2 * * *");
My EntryJob interface having RunAsync method.
public class EntryJob: IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
public async Task RunAsync(int Id, SomeObj obj)
{
//Some coe
}
}
And interface look like this
[DisableConcurrentExecution(3600)] <-- Tried here
public interface IJob
{
[DisableConcurrentExecution(3600)] <-- Tried here
Task RunAsync(int Id, SomeObj obj);
}
Now I want to prevent RunAsync method to call multiple times if Id is same. I have tried to put DisableConcurrentExecution on top of the RunAsync method at both location inside interface declaration and also from where Interface is implemented.
But it seems like not working for me. Is there any way to prevent concurrency based on Id?
The existing implementation of DisableConcurrentExecution does not support this. It will prevent concurrent executions of the method with any args. It would be fairly simple to add support in. Note below is untested pseudo-code:
public class DisableConcurrentExecutionWithArgAttribute : JobFilterAttribute, IServerFilter
{
private readonly int _timeoutInSeconds;
private readonly int _argPos;
// add additional param to pass in which method arg you want to use for
// deduping jobs
public DisableConcurrentExecutionAttribute(int timeoutInSeconds, int argPos)
{
if (timeoutInSeconds < 0) throw new ArgumentException("Timeout argument value should be greater that zero.");
_timeoutInSeconds = timeoutInSeconds;
_argPos = argPos;
}
public void OnPerforming(PerformingContext filterContext)
{
var resource = GetResource(filterContext.BackgroundJob.Job);
var timeout = TimeSpan.FromSeconds(_timeoutInSeconds);
var distributedLock = filterContext.Connection.AcquireDistributedLock(resource, timeout);
filterContext.Items["DistributedLock"] = distributedLock;
}
public void OnPerformed(PerformedContext filterContext)
{
if (!filterContext.Items.ContainsKey("DistributedLock"))
{
throw new InvalidOperationException("Can not release a distributed lock: it was not acquired.");
}
var distributedLock = (IDisposable)filterContext.Items["DistributedLock"];
distributedLock.Dispose();
}
private static string GetResource(Job job)
{
// adjust locked resource to include the argument to make it unique
// for a given ID
return $"{job.Type.ToGenericTypeString()}.{job.Method.Name}.{job.Args[_argPos].ToString()}";
}
}
Is it normal that a client application took a longer time to insert data into GemFire Cluster for the first time? For example, my client application took around 4 seconds to insert GemFire Cluster successfully. However , the subsequent insert only took around 1 second. May i know what is the reason behind it?
#Configuration
#EnablePool(name = "sgpool", socketBufferSize = 1000000, prSingleHopEnabled = true, idleTimeout = 10000)
public class RegionConfiguration {
#Bean("People")
public ClientRegionFactoryBean<String, Person> customersRegion(GemFireCache gemfireCache) {
ClientRegionFactoryBean<String, Person> customersRegion = new ClientRegionFactoryBean<>();
customersRegion.setCache(gemfireCache);
customersRegion.setClose(false);
customersRegion.setPoolName("sgpool");
customersRegion.setShortcut(ClientRegionShortcut.CACHING_PROXY);
return customersRegion;
}
}
#ClientCacheApplication
#EnablePdx
#Service
#EnableGemfireRepositories(basePackageClasses = PersonRepository.class)
#Import(RegionConfiguration.class)
public class PersonDataAccess {
#Autowired
#Qualifier("People")
private Region<String, Person> peopleRegion;
#Autowired
private PersonRepository personRepository;
#PostConstruct
public void init() {
peopleRegion.registerInterestForAllKeys();
}
public void saveAll(Iterable<Person> iteratorList) {
personRepository.saveAll(iteratorList);
}
}
#Service
#EnableScheduling
#Log4j2
public class PersonService {
#Autowired
public PersonDataAccess personDataAccess;
private Person createPerson(String ic, int age) {
return new Person(ic, "Jack - " + age, "Kay - " + age, LocalDate.of(2000, 12, 10), age, 2,
new Address("Jack Wonderful Land 12", "Wonderful Land", "Wonderful 101221"), "11111", "22222", "33333",
"Dream", "Space");
}
#Scheduled(fixedDelay = 1000, initialDelay = 5000)
public void testSaveRecord() {
ArrayList<Person> personList = new ArrayList<>();
for (int counter = 0; counter < 30000; counter++) {
personList.add(createPerson("S2011" + counter, counter));
}
log.info("start saving person");
personDataAccess.saveAll(personList);
log.info("Complete saving all the message");
}
}
GemFire Configuration:
Using PDX Serialization
1 Locator and Cache Server (GemFire 9.10.5
Partition Region
Client Application (Using Spring Data GemFire 2.3.4)
Thank you so much
Clearly the 1st time there’s a PDX cache register process going on between client and server kind of like
client server event distribution
The register process does not need to happen 2nd time because it’s now setup.
The PDX registration process is explained more here where it says: Geode maintains a central registry of the PDX domain object metadata.
There must be a way to get the region setup done before your 1st insert, so that each insert is the same.
Maybe this could help
1. Is it possible to put non-POJO class instances as the value of a cache?
For example, I have a QueryThread class which is a subclass of java.lang.Thread and I am trying to put this instance in a cache. It looks like the put operation is failing because this cache is always empty.
Consider the following class:
public class QueryThread extends Thread {
private IgniteCache<?, ?> cache;
private String queryId;
private String query;
private long timeIntervalinMillis;
private volatile boolean running = false;
public QueryThread(IgniteCache<?, ?> dataCache, String queryId, String query, long timeIntervalinMillis) {
this.queryId = queryId;
this.cache = dataCache;
this.query = query;
this.timeIntervalinMillis = timeIntervalinMillis;
}
public void exec() throws Throwable {
SqlFieldsQuery qry = new SqlFieldsQuery(query, false);
while (running) {
List<List<?>> queryResult = cache.query(qry).getAll();
for (List<?> list : queryResult) {
System.out.println("result : "+list);
}
System.out.println("..... ");
Thread.sleep(timeIntervalinMillis);
}
}
}
This class is not a POJO. How do I store an instance of this class in the cache?
I tried implementing Serializable (didn't help).
I need to be able to do this:
queryCache.put(queryId, queryThread);
Next I tried broadcasting the class using the IgniteCallable interface. But my class takes multiple arguments in the constructor. I feel PeerClassLoading is easy if the class takes a no-arg constructor:
IgniteCompute compute = ignite.compute(ignite.cluster().forServers());
compute.broadcast(new IgniteCallable<MyServiceImpl>() {
#Override
public MyServiceImpl call() throws Exception {
MyServiceImpl myService = new MyServiceImpl();
return myService;
}
});
2. How do I do PeerClassLoading in the case of a class with multi-arg constructor?
It's restricted to put Thread instances to the cache, Thread instance cannot be serialized due to call to Native Methods. Thats why you always get empty value.
PeerClassLoading is a special distributed ClassLoader in Ignite for inter-node byte-code exchange. So, it's only about sharing classes between nodes. It doesn't make sense how many arguments in constructor class have.
But, on the other hand, object, that you created, will be serialised and sent to other nodes and for deserialisation it will need a default(non-arg) constructor.
Hi I am new to Storm and Kafka.
I am using storm 1.0.1 and kafka 0.10.0
we have a kafkaspout that would receive java bean from kafka topic.
I have spent several hours digging to find the right approach for that.
Found few articles which are useful but none of the approaches worked for me so far.
Following is my codes:
StormTopology:
public class StormTopology {
public static void main(String[] args) throws Exception {
//Topo test /zkroot test
if (args.length == 4) {
System.out.println("started");
BrokerHosts hosts = new ZkHosts("localhost:2181");
SpoutConfig kafkaConf1 = new SpoutConfig(hosts, args[1], args[2],
args[3]);
kafkaConf1.zkRoot = args[2];
kafkaConf1.useStartOffsetTimeIfOffsetOutOfRange = true;
kafkaConf1.startOffsetTime = kafka.api.OffsetRequest.LatestTime();
kafkaConf1.scheme = new SchemeAsMultiScheme(new KryoScheme());
KafkaSpout kafkaSpout1 = new KafkaSpout(kafkaConf1);
System.out.println("started");
ShuffleBolt shuffleBolt = new ShuffleBolt(args[1]);
AnalysisBolt analysisBolt = new AnalysisBolt(args[1]);
TopologyBuilder topologyBuilder = new TopologyBuilder();
topologyBuilder.setSpout("kafkaspout", kafkaSpout1, 1);
//builder.setBolt("counterbolt2", countbolt2, 3).shuffleGrouping("kafkaspout");
//This is for field grouping in bolt we need two bolt for field grouping or it wont work
topologyBuilder.setBolt("shuffleBolt", shuffleBolt, 3).shuffleGrouping("kafkaspout");
topologyBuilder.setBolt("analysisBolt", analysisBolt, 5).fieldsGrouping("shuffleBolt", new Fields("trip"));
Config config = new Config();
config.registerSerialization(VehicleTrip.class, VehicleTripKyroSerializer.class);
config.setDebug(true);
config.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(args[0], config, topologyBuilder.createTopology());
// StormSubmitter.submitTopology(args[0], config,
// builder.createTopology());
} else {
System.out
.println("Insufficent Arguements - topologyName kafkaTopic ZKRoot ID");
}
}
}
I am serializing the data at kafka using kryo
KafkaProducer:
public class StreamKafkaProducer {
private static Producer producer;
private final Properties props = new Properties();
private static final StreamKafkaProducer KAFKA_PRODUCER = new StreamKafkaProducer();
private StreamKafkaProducer(){
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "com.abc.serializer.MySerializer");
producer = new org.apache.kafka.clients.producer.KafkaProducer(props);
}
public static StreamKafkaProducer getStreamKafkaProducer(){
return KAFKA_PRODUCER;
}
public void produce(String topic, VehicleTrip vehicleTrip){
ProducerRecord<String,VehicleTrip> producerRecord = new ProducerRecord<>(topic,vehicleTrip);
producer.send(producerRecord);
//producer.close();
}
public static void closeProducer(){
producer.close();
}
}
Kyro Serializer:
public class DataKyroSerializer extends Serializer<Data> implements Serializable {
#Override
public void write(Kryo kryo, Output output, VehicleTrip vehicleTrip) {
output.writeLong(data.getStartedOn().getTime());
output.writeLong(data.getEndedOn().getTime());
}
#Override
public Data read(Kryo kryo, Input input, Class<VehicleTrip> aClass) {
Data data = new Data();
data.setStartedOn(new Date(input.readLong()));
data.setEndedOn(new Date(input.readLong()));
return data;
}
I need to get the data back to the Data bean.
As per few articles I need to provide with a custom scheme and make it part of topology but till now I have no luck
Code for Bolt and Scheme
Scheme:
public class KryoScheme implements Scheme {
private ThreadLocal<Kryo> kryos = new ThreadLocal<Kryo>() {
protected Kryo initialValue() {
Kryo kryo = new Kryo();
kryo.addDefaultSerializer(Data.class, new DataKyroSerializer());
return kryo;
};
};
#Override
public List<Object> deserialize(ByteBuffer ser) {
return Utils.tuple(kryos.get().readObject(new ByteBufferInput(ser.array()), Data.class));
}
#Override
public Fields getOutputFields( ) {
return new Fields( "data" );
}
}
and bolt:
public class AnalysisBolt implements IBasicBolt {
/**
*
*/
private static final long serialVersionUID = 1L;
private String topicname = null;
public AnalysisBolt(String topicname) {
this.topicname = topicname;
}
public void prepare(Map stormConf, TopologyContext topologyContext) {
System.out.println("prepare");
}
public void execute(Tuple input, BasicOutputCollector collector) {
System.out.println("execute");
Fields fields = input.getFields();
try {
JSONObject eventJson = (JSONObject) JSONSerializer.toJSON((String) input
.getValueByField(fields.get(1)));
String StartTime = (String) eventJson.get("startedOn");
String EndTime = (String) eventJson.get("endedOn");
String Oid = (String) eventJson.get("_id");
int V_id = (Integer) eventJson.get("vehicleId");
//call method getEventForVehicleWithinTime(Long vehicleId, Date startTime, Date endTime)
System.out.println("==========="+Oid+"| "+V_id+"| "+StartTime+"| "+EndTime);
} catch (Exception e) {
e.printStackTrace();
}
}
but if I submit the storm topology i am getting error:
java.lang.IllegalStateException: Spout 'kafkaspout' contains a
non-serializable field of type com.abc.topology.KryoScheme$1, which
was instantiated prior to topology creation.
com.minda.iconnect.topology.KryoScheme$1 should be instantiated within
the prepare method of 'kafkaspout at the earliest.
Appreciate help to debug the issue and guide to right path.
Thanks
Your ThreadLocal is not Serializable. The preferable solution would be to make your serializer both Serializable and threadsafe. If this is not possible, then I see 2 alternatives since there is no prepare method as you would get in a bolt.
Declare it as static, which is inherently transient.
Declare it transient and access it via a private get method. Then you can initialize the variable on first access.
Within the Storm lifecycle, the topology is instantiated and then serialized to byte format to be stored in ZooKeeper, prior to the topology being executed. Within this step, if a spout or bolt within the topology has an initialized unserializable property, serialization will fail.
If there is a need for a field that is unserializable, initialize it within the bolt or spout's prepare method, which is run after the topology is delivered to the worker.
Source: Best Practices for implementing Apache Storm
I just read some source code is from org.apache.cxf.common.logging.JDKBugHacks and also in
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/core/JreMemoryLeakPreventionListener.java. In order to make my question clear not too broad. :)
I just ask one piece of code in them.
// Calling getPolicy retains a static reference to the context
// class loader.
try {
// Policy.getPolicy();
Class<?> policyClass = Class
.forName("javax.security.auth.Policy");
Method method = policyClass.getMethod("getPolicy");
method.invoke(null);
} catch (Throwable e) {
// ignore
}
But I didn't understand this comment. "Calling getPolicy retains a static reference to the context class loader". And they trying to use JDKBugHacks to work around it.
UPDATE
I overlooked the static block part. Here it is. This is the key. Actually it already has policy cached. So why cache contextClassLoader also? In comment, it claims #deprecated as of JDK version 1.4 -- Replaced by java.security.Policy.
I have double checked the code of java/security/Policy.java. It really removed the cached classloader. So my doubt is valid! :)
#Deprecated
public abstract class Policy {
private static Policy policy;
private static ClassLoader contextClassLoader;
static {
contextClassLoader = java.security.AccessController.doPrivileged
(new java.security.PrivilegedAction<ClassLoader>() {
public ClassLoader run() {
return Thread.currentThread().getContextClassLoader();
}
});
};
I also add the getPolicy source code.
public static Policy getPolicy() {
java.lang.SecurityManager sm = System.getSecurityManager();
if (sm != null) sm.checkPermission(new AuthPermission("getPolicy"));
return getPolicyNoCheck();
}
static Policy getPolicyNoCheck() {
if (policy == null) {
synchronized(Policy.class) {
if (policy == null) {
String policy_class = null;
policy_class = java.security.AccessController.doPrivileged
(new java.security.PrivilegedAction<String>() {
public String run() {
return java.security.Security.getProperty
("auth.policy.provider");
}
});
if (policy_class == null) {
policy_class = "com.sun.security.auth.PolicyFile";
}
try {
final String finalClass = policy_class;
policy = java.security.AccessController.doPrivileged
(new java.security.PrivilegedExceptionAction<Policy>() {
public Policy run() throws ClassNotFoundException,
InstantiationException,
IllegalAccessException {
return (Policy) Class.forName
(finalClass,
true,
contextClassLoader).newInstance();
}
});
} catch (Exception e) {
throw new SecurityException
(sun.security.util.ResourcesMgr.getString
("unable to instantiate Subject-based policy"));
}
}
}
}
return policy;
}
Actually I dig deeper, I find some interesting thing. Someone report a bug to apache CXF about the org.apache.cxf.common.logging.JDKBugHacks for this piece code recently.
In order for disabling url caching, JDKBugHacks runs:
URL url = new URL("jar:file://dummy.jar!/");
URLConnection uConn = url.openConnection();
uConn.setDefaultUseCaches(false);
When having the java.protocol.handler.pkgs system property set, that can lead to deadlocks between the system classloader and the file protocol Handler in particular situations (for instance if the file protocol URLStreamHandler is a signleton).
Besides that, the code above is really there for the sake of setting defaultUseCaches to false only, so actually opening a connection can be avoided, to speed up the execution.
So the fix is
URL url = new URL("jar:file://dummy.jar!/");
URLConnection uConn = new URLConnection(url) {
#Override
public void connect() throws IOException {
// NOOP
}
};
uConn.setDefaultUseCaches(false);
It's normal that JDK or apache cxf to have some minor bugs. And normally they will fix it.
javax.security.auth.login.Configuration has the same issues with Policy but it's not Deprecated.
The Policy class in java 6 contains a static reference to a classloader that is initialized to the current threads context classloader on the first access to the class:
private static ClassLoader contextClassLoader;
static {
contextClassLoader =
(ClassLoader)java.security.AccessController.doPrivileged
(new java.security.PrivilegedAction() {
public Object run() {
return Thread.currentThread().getContextClassLoader();
}
});
};
Tomcats lifecycle listener is making sure to to initialize this class from within a known environment where the context classloader is set to the system classloader. If this class was first accessed from within a webapp, it would retain a reference to the webapps classloader. This would prevent the webapps classes from getting garbage collected, creating a leak of perm gen space.