I have 3 virtual machines that I want to add to an Ambari cluster.
I'm going through the setup wizard to do this.
My VMs have less than 2GB of space in the drive mounted on /
Ambari complains about this.
Is there any way to tell Ambari that I want it to use a different location?
I would like to tell Ambari to use the /data location for each host.
Any help will be greatly appreciated.
Assuming you're attempting to install an HDP stack. No. You will need more space on the root partition. All of the services (Spark, HBase, HDFS, Yarn, Oozie, etc.) are installed into /usr/hdp/<version>. This is not overridable since each of these services is installed using an rpm and the default location in which those rpms install into is hard coded throughout several of the service descriptors that help provision the cluster.
Related
I'm trying to find a viable solution using Redis as a Master/Slave(at least 2 slaves) configuration. I have Docker containers with Ubuntu 16.04 OS & Redis server/sentinel installed (latest stable).
I'm not looking for a clustered setup. I would like to have the master redis db on one pod, and the slaves on their own pod (all three will be on separate vm's or physical boxes). I'll want to use yaml/Kubernetes nodeSelector to assign where they can spin up.
From my research, it appears I want to run Redis Sentinel services on each pod as well. They key here as well is I want to specify where each Master/Slave POD can run. I've investigated https://github.com/kubernetes/kubernetes/tree/master/examples/redis but that does not give me the control I want. Maybe Redis 4.x helps, but I can't find any examples. Any pointers would be appreciated. I've searched all over this site for an answer w/o any luck.
In my lab, I am currently managing a 20 nodes cluster with Cobbler and Chef. Cobbler is used for OS provisioning and basic network settings, which is working fine as expected. I can manage several OS distributions with preseed-based NQA installation and local repo mirroring.
We also successfully installed chef server and started managing nodes but chef is not working as I expected. The issue is that I am not being able to set node dependencies within chef. Our one important use case is this:
We are setting up ceph and openstack on these nodes
Ceph should be installed before openstack because openstack uses ceph as back-end storage
Ceph monitor should be installed before Ceph osd because creating osd requires talking to monitor
The dependencies between Openstack and Ceph does not matter because it is a dependency in one node; just installing openstack later would resolve the issue.
However, a problem arises with the dependency between ceph monitor and ceph osd. Ceph osd provisioning requires a running ceph monitor. Therefore, ceph osd recipe should always be run after ceph mon recipe finishes in another node. Our current method is just to run "chef-client" in "ceph-osd" node after "chef-client" run completely finishes in "ceph-mon" node but I think this is a too much of a hassle. Is there a way to set these dependencies in Chef so that nodes will provision sequentially according to their dependencies? If not, are there good frameworks who handles this?
In chef itself, I know no method for orchestrating (that's not chef Job).
A workaround given your use case could be to use tags and search.
You monitor recipe could tag the node at end (with tag("CephMonitor") or with setting any attribute you wish to search on).
After that the solr index of chef has to catch it up (usually in the minute) and you can use search in the Cephosd recipe you can do something like this:
CephMonitor = search(:node,"tags:CephMonitor") || nil
return if CephMonitor.nil?
[.. rest of the CephOsd recipe, using the CephMonitor['fqdn'] or other attribute from the node ..]
The same behavior can be used to avoid trying to run the OpenStack recipe until the osd has run.
The drawback if that it will take 2 or 3 chef run to get to a converged infrastructure.
I've nothing to recommend to do the orchestration, zookeeper or consul could help instead of tags and to trigger the runs.
Rundeck can tage the runs on different nodes and aggregate this in one job.
Which is best depends on your feeling there.
I have a weblogic 12 cluster. Files get pushed to it both through http forms and through scp to a single machine on the cluster. However I need the files on all the nodes of the cluster. I can run scp myself and copy to all parts of the cluster, but I was hoping that weblogic supported the functionality in some manner. I don't have a disk shared between the machines that would make this easier. Nor can I create a shared disk.
Does anybody know?
No There is no way for WLS to ensure a file copied on one instance of WLS is copied to another. Especially when you are copying it over even using scp.
Please use a shared storage mount, so that all managed servers can refer to this location with out the need to do SCP.
I have a small hadoop cluster i.e one master and three slave nodes. I have to monitor cluster. I have found that we can use Ambari. CentOS 7 is installed on all machines. Please provide a complete details how I can do that ?. I have found that Ambari can be used for new cluster i.e you have to install new cluster. It does not work with already running cluster?
At the moment Ambari does not support CentOS 7, so that's not going to work.
However, Ambari does not perform cluster monitoring on its own. It uses Nagios for the purpose. Nagios is an independent software project that you can setup independently. That said it's kinda painful to do.
ambari-server for Ambari 2.2+ can be installed and works good on CentOS 7.
You have to installed ambari-server on one of the hosts (master node) and can use the webUI hostname:8080 for installing ambari agents on other hosts. Alternatively, ambari agents can be installed manually on other hosts can can be linked to communicate with the ambari-server.
Main OS is windows 7 64bit. Using VM player to create two vm CentOS 5.6 system. The net connection is bridge. I installed Hbase on both of the CentOS system， one is master, the other is slave. When I enter the shell, and run status 'details'.
The error from master is
zookeeper.ZKConfig: no valid quorum servers found in zoo.cfg ERROR:
org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is
preventing HBase from connecting to ZooKeeper
And the error from slave is
ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is
the default). Consider inspecting your ZK server logs for that error
and then make sure you are reusing HBaseConfiguration as often as you
can. See HTable's javadoc for more information.
Please give me some suggestion.
Thanks a lot
Check if this is within your .bashrc, if not, add them and restart all hbase services (do not forget to manually run them as well), that did it for me with a pseudo-distributed installation. My problem (and maybe yours as well) was that Hbase wasn't detecting it's configuration.
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HBASE_CONF_DIR=/etc/hbase/conf
I see this very often on my machine. I don't have a failsafe cure, but end up running stop-all.sh, and deleting every place that hadoop and dfs (its a dfs failure) store their temp files. It seems to happen after my computer goes to sleep while dfs is running.
I am going to experiment with single-user mode to avoid this. I dont need distribution while developing.