After a recent upgrade of the gridgain control center , we lose cluster info and credentials everytime the pod restarts.
Our gridgain control center backend is a deployment and not a stateful set and we were using persistent volumes to save the state.
Can someone help me understand how/where the cluster and credentials are stored in the control center. This will help us debug the issue.
In the backend you need a persistent volume for /opt/gridgain-control-center/work. Using a stateful set is a good idea here.
Related
We are running one of our services in a newly created kubernetes cluster. Because of that, we have now switched them from the previous "in-memory" cache to a Redis cache.
Preliminary tests on our application which exposes an API shows that we experience timeouts from our applications to the Redis cache. I have no idea why and it issue pops up very irregularly.
So I'm thinking maybe the reason for these timeouts are actually network related. Is it a good idea to put in affinity so we always run the Redis-cache on the same nodes as the application to prevent network issues?
The issues have not arisen during "very high load" situations so it's concerning me a bit.
This is an opinion question so I'll answer in an opinionated way:
Like you mentioned I would try to put the Redis and application pods on the same node, that would rule out wire networking issues. You can accomplish that with Kubernetes pod affinity. But you can also try nodeslector, that way you always pin your Redis and application pods to a specific node.
Another way to do this is to taint your nodes where you want to run your workloads and then add a toleration to the Redis and your application pods.
Hope it helps!
We are using prometheus in our production envirment recently. Before we only have 30-40 nodes for each service and those servers not change very often, so we just write it in the prometheus.yml, but right now it become too long to hold in one file and change much frequently then before, so my question is should i use file_sd_config to put those server list out of yml file and change those config files sepearately, or using consul for service discovery(same much easy to handle changes).
I have install 3 nodes consul cluster in data center and as i can see if i change to use consul to slove this problem , i also need to install consul client in each server(node) and define its services info. Is that correct? or does anyone have good advise.
Thanks
I totally advocate the use of a service discovery system. It may be a bit hard to deploy at first but surely it will worth it in the future.
That said, Prometheus comes with a lot of service discovery integrations. It's possible that you don't need a Consul cluster. If your servers are in a cloud provider like AWS, GCP, Azure, Openstack, etc, prometheus are able to autodiscover the instances.
If you keep running with Consul, the answer is yes, the agent must be running in every node. You can also register services and nodes via API but it's easier to deploy the agent.
I have a Python app running on a Kubernetes Cluster. I want to get app specific monitoring information e.g. logging info that I have in my app (using python logging module) and also metrics info that I am collecting using collectd.
I understand Operations Management Suite can be used to monitor the cluster itself but can it also provide access to app specific logs and metrics?
Appreciate any pointers on how to do this.
Thanks
Rajeev
First I would like to thanks a lot for this forum.
I have a doubt about the cluster configuration with Hazelcast and Load Balance.
In the documentation https://docs.wso2.com/display/CLUSTER420/Clustering+the+Gateway
in section load balance configuration appear:
upstream wso2.am.com {
**sticky cookie JSESSIONID;**
server xxx.xxx.xxx.xx4:9763;
server xxx.xxx.xxx.xx5:9763;
}
Why to use sticky if the cluster already make the session control?
My understanding is wrong?
Thanks a lot.
In APIM, hazelcast clustering is used for cache invalidation among nodes, inter-node communication for dep-sync etc, but not for session replication. Therefore you need sticky sessions.
I have developed an application and want to deploy it on a cluster environment.But I am not sure how replicate session when one server goes down.
What are the things I need to do for session replication?
Any suggestion would be greatly appreciated!
Try using this
It is always better to have session replication off of weblogic and on the web tier end as the target is weblogic.
You can also look into session persistence and how it can be achieved is hazelcast.
I'm sure a proper cluster with caching such as coherence can maintain sessions over multiple machines and provide high availability in weblogic. From an infrastructure stand point, I would go with Coherence or "Coherence-like" products to achieve session replication and persistence.
WebLogic has built-in clustering, so you don't need some other thing to do this. Also, WebLogic Suite includes Coherence, and with WebLogic Suite you can turn on Coherence session clustering just by clicking a check-box in the console.
Please read this