I am using an AWS EMR cluster with Alluxio installed n every node. I want to now deploy Alluxio in High Availability.
https://docs.alluxio.io/os/user/stable/en/deploy/Running-Alluxio-On-a-HA-Cluster.html#start-an-alluxio-cluster-with-ha
I am following the above documentation, and see that "On all the Alluxio master nodes, list all the worker hostnames in the conf/workers file, and list all the masters in the conf/masters file".
My concern is that since I have an AWS-managed scaling cluster the worker nodes keep added and removed based on cluster loads. How can I maintain a list of constant masters and workers under conf/masters and conf/workers in a managed Scaling cluster?
this conf/workers and conf/masters conf file is only used for intiial setup through scripts. Once the cluster is running, you don’t need to update them any more.
E.g., in an say EMR cluster, you can add a new slave node as Alluxio worker and as long as you specify the correct Alluxio master address, this new Alluxio worker will be able to register itself and serve in the fleet like other workers,
Related
I have a quick question about redis cluster.
I'm setting up a redis cluster on google cloud kubernetes engine. I'm using the n1-highmem-2 machine type with 13GB RAM, but I'm slightly confused how to calculate the total available size of the cluster.
I have 3 nodes with each 13GB ram. I'm running 6 pods (2 on each node), 1 master and 1 slave per node. This all works. I've assigned 6GB of RAM to each pod in my pod definition yaml file.
Is it correct to say that my total cluster size would be 18GB (3 masters * 6GB), or can I count the slaves size with the total size of the redis cluster?
Redis Cluster master-slave model
In order to remain available when a subset of master nodes are failing or are not able to communicate with the majority of nodes, Redis Cluster uses a master-slave model where every hash slot has from 1 (the master itself) to N replicas (N-1 additional slaves nodes).
So, slaves are replicas(read only) of masters(read-write) for availability, hence your total workable size is the size of your master pods.
Keep in mind though, that leaving masters and slaves on the same Kubernetes node only protects from pod failure, not node failure and you should consider redistributing them.
You didn't mention how are you installing Redis, But I'd like to mention Bitnami Redis Helm Chart as it's built for use even on production and deploys 1 master and 3 slaves providing good fail tolerance and have tons of configurations easily personalized using the values.yaml file.
I am trying to set up hive using mr3 on a kubernetes cluster hosted on AWS ec2. When I run the command run-hive.sh, Hive-server starts and the master-DAg is initialised but then it gets stuck on pending. When I describe the pod. This is the error message shows. I have kept the resources to minimum so it should not be that issue and I do not have any tainted nodes. If you know any alternative for running hive on Kubernetes with access to S3 or a better way to implement mr3 hive on Kubernetes cluster, please share.
One of the node description
Based on the topic i think the problem here is your cluster have not enough resources on your worker nodes, and a master node is tainted.
So the option here is either inreasing the resources on workers or taint the master node so You would be able to schedule pods there.
Control plane node isolation
By default, your cluster will not schedule pods on the control-plane node for security reasons. If you want to be able to schedule pods on the control-plane node, e.g. for a single-machine Kubernetes cluster for development, run:
kubectl taint nodes --all node-role.kubernetes.io/master-
This will remove the node-role.kubernetes.io/master taint from any nodes that have it, including the control-plane node, meaning that the scheduler will then be able to schedule pods everywhere
I have a setup with single Redis node and I use a list to push and consume tasks through various clients connected to this single Redis node.
I want to move this setup to Redis cluster eventually. Should the task queue/list which is present in the above setup be split across all the nodes of Redis cluster or should it fit in only one of the hash slots or in other words in one of the redis nodes in the cluster.
When running a single instance redis, I can use "slave of" to create a (or as many I like) readonly replica of this one redis node.
When using redis cluster, I split my Data into Partitons (Masters) and can create a slave for each partition.
Is it possible to treat this cluster as a single instance and connect a "slave of" Slave to this cluster which will hold a replica of all Data in the cluster and not just the partition of the connected node?
If not possible with redis cluster, is this might a working solution when using sentinel?
Our current Problem:
We are using the "slave of" feature together with keepalived to failover our redis instance on an outage of the master.
But we have lots of "slave of" slaves connected to the virtual IP of the failover setup, to deliver cached data.
Now everytime the system fails over (for maintenance reasons e.g.) all connected slaves have a timout for up to 30 seconds, when they have to resync their data with the new master.
We allready played with all possible redis config parameters but can't get this syncing time to be shorter (e.g. by relying on the replication-backlog, which isn't available on the new master after the failover).
Anyone any ideas?
a very good doc here : http://redis.io/presentation/Redis_Cluster.pdf and here http://fr.slideshare.net/NoSQLmatters/no-sql-matters-bcn-2014 (slide #9) or better https://www.javacodegeeks.com/2015/09/redis-clustering.html
If you want "slave" in Redis cluster mode, you need use replication of all nodes.
Regards,
Well, I just read this article:
https://seanmcgary.com/posts/how-to-build-a-fault-tolerant-redis-cluster-with-sentinel
The author used a single master with Redis Cluster, with 2 slaves per master, instead of one, and he let Redis Sentinel take care of the election of a slave to a master when the master is down.
You could play with this setup to see if the election of Master occurs quickly. While it's happening, clients would be served by a slave and should experience no downtime.
I have 4 systems. I want to crawl some data. For that first I need to configure cluster. I am confused about placement of components.
should I place all component (hadoop, hive, hbase, nutch) in one machine and add other machines as nodes in hadoop?
Should I place hbase in one machine, nutch in other and hadoop in third and add forth machine as slave of hadoop?
Should HBase be in pseudo distributed mode or full distributed.
How many slaves I sholud add in hbase if I run it as fully distributed mode.
What should be the best way. PLease guide step by step ( For hbase and hadoop)
Say you have 4 nodes n1, n2, n3 and n4.
You can install hadoop and hbase in distributed mode.
If you are using Hadoop 1.x -
n1 - hadoop master[Namenode and Jobtracker]
n2, n3 and n3 - hadoop slaves [datanodes and tasktrackers]
For HBase, you can choose n1 or any other node as Master node, Since Master node are usually not CPU/Memory intensive, all Masters can be deployed on single node on test setup, However in Production its good to have each Master deployment on a separate node.
Lets say n2 - HBase Master, remaining 3 nodes can act as regionservers.
Hive and Nutch can reside on any node.
Hope this helps; For a test setup this should be good to go.
Update -
For Hadoop 2.x, since your cluster size is small, Namenode HA deployment can be skipped.
Namenode HA would require two nodes one each for an active and standby node.
A zookeeper quorum which again requires odd number of nodes so a minimum of three nodes would be required.
A journal quorum again require a minimum of 3 nodes.
But for a cluster this small HA might not be a major concern. So you can keep
n1 - namenode
n2 - ResouceManager or Yarn
and remaining nodes can act as datanodes, try not to deploy anything else on the yarn node.
Rest of the deployment for HBase, Hive and Nutch would remain same.
In my opinion, you should install Hadoop in fully distributed mode, so the jobs could run in parallel manner and much faster, as the MapReduce tasks will be distributed in 4 machines. Of course, the Hadoop's master node should run in one single machine.
If you need to process big amount of data, it's a good choice to install HBase in one single machine and the Hadoop in 3.
You could make all the above very easy using tools/platforms with a very friendly GUI like Cloudera Manager and Hortonworks. They will help you to control and maintain your cluster better but they are also provide Health Monitoring, Cluster Analytics as well as E-Mail notifications for every error occurs in your cluster.
Cloudera Manager
http://www.cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise/cloudera-manager.html
Hortonworks
http://hortonworks.com/
In these two links, you can find more guidance about how you could costruct your cluster