Are the control scripts (start-dfs.sh and start-mapred.sh) used by CDH to start daemons on the fully distributed cluster?
I downloaded and installed CDH5, but could not see the control scripts in the installation, and wondering how does CDH start the daemons on slave nodes?
Or since the daemons are installed as services, they do start with the system start-up. Hence there is no need for control scripts in CDH unlike apache hadoop.
Not as such, no. If I recall correctly, Cloudera Manager invokes the daemons using Supervisor (http://supervisord.org/). So, you manage the services in CM. CM itself runs agent processes as a service on each node, and you can find its start script in /etc/init.d. There is no need for you to install or start or stop anything. You install, deploy config, control and monitor servies in CM.
Related
I have created a multi-job with few Sub-Jobs (which run my each test suite), in the main job i am cloning my repo from git and store it jenkins/workspace in a linux machine. all my sub jobs run in windows nodes. is there any way to share or access my linux machines jenkins/workspace with other jobs running on windows node. Please suggest if there is any other way to acheive it.
I am trying to get a simple, single instance of RabbitMQ running via this RabbitMQ-chef recipe. Essentially, I have an Ubuntu machine, and I’d like Chef to take care of all the heavy lifting necessary to install, configure and get RabbitMQ running on it.
The documentation isn’t really written for beginners, but I went ahead and started by downloading the cookook:
knife cookbook site download rabbitmq
WARNING: No knife configuration file found
Downloading rabbitmq from the cookbooks site at version 4.1.2 to /Users/myuser/sandbox/chef/0.6.2/rabbitmq-4.1.2.tar.gz
Cookbook saved: /Users/myuser/sandbox/chef/0.6.2/rabbitmq-4.1.2.tar.gz
However now I’m at a complete loss as to what I need to script/configure in order to add this cookbook to my Chef server, such that when I SSH onto my target Ubuntu machine, I can bootstrap that machine with Chef and then run something to turn the Ubuntu machine into a RabbitMQ server. Any ideas?
Im trying to setup Jenkins to run tests on a Virtual Machine but im not to sure how to proceed.
What id like to be able to do is to get Jenkins to build the environment on the vm and then have Jenkins execute the test scripts on the vm environment. After the tests have passed/failed id then like Jenkins to clean the database and pull down the virtual environment.
Server box - Windows 7
Virtaul Machine - VMWare
So what im looking for is some information or tutorials on how to implement the above. It would also be helpful if you could recommend what Jenkins plugins I can use to implement the above or if you want to go above and beyond can you outline the steps needed to achieve the above.
Any help would be appreciated.
I'm doing just that in my environment using the vSphere Cloud Plugin. Here's a basic step-by-step guide:
Install the plugin
Configure your ESX/ESXi server as a new "vSphere Cloud"
Create a new Jenkins node, of type "Slave virtual computer running under vSphere Cloud" (which becomes available after Installing the plugin).
When configuring the new node, optionally specify a snapshot name. This will revert the VM to this snapshot when the node launches.
Use the node in a pipeline script: node("node-name-or-label") { ...your code here... }
I use the method above with about 10 Windows nodes, reverting each to a "Clean" snapshot to start each build with a known state.
I just followed the Hadoop(0.20.2) installation tutorial and did the set up. I can run map reduce program on the cluster through eclipse. Now my problem is how can I connect to Hadoop clusters from my local system. Local system is windows 7 and I have installed eclipse plugin for Hadoop. I was trying to connect to Hadoop from my local system which is windows(My local system and Hadoop system are in same subnet). I got connection timed out error while connecting to Hadoop server.
In configuration files of Hadoop I have given actual IP addresses.
Not sure which step I have missed out?
I recently read, that the eclipse plugin won't work at all. But you can simply connect to your Cluster with the configuration keys:
mapred.job.tracker
fs.default.name
EDIT: here is a working version Apache Jira: Eclipse Plugin does not work with Eclipse Ganymede (3.4)
Is is possible to automate the installation of an OS using VMware or any other virtualization product?
One of our products consists of a customized version of CentOS that installs the OS and our application on a server. It's much like any CentOS/RHEL installation where you choose a mode that corresponds to different kickstart options, and then you choose your keyboard type. The rest of the installation is automatic.
What I'd like to have is an automated system that will create a new guest VM, boot it with the ISO image of our product, start the installation (including choosing the keyboard), wait for the reboot, and then launch a set of automated tests.
I know that there are plenty of ways to automate the creation of new VM guests from existing templates/images, and I know you can use the VIX API to interact with virtual machines, but the VIX API seems to require that VMware tools is already running (which won't be the case when you're booting from the CentOS install disk).
This answer (Automating VMWare or VirtualPC) indicates that you can script VMware to boot from an ISO that does an unattended installation, but I would really like to test the same process that our customers will be using.
Another option might be to use Xen's fully-virtualized mode and see if scripting it over the serial port will work.
TIA,
Jason
I have a very very similar question, it is on superuser:
https://superuser.com/questions/36047/moving-vmware-os-image-as-primary-os-on-a-system
You can also use VirtualBox instead of VMWare. The VirtualBox SDK allows you to directly control the keyboard, the mouse the serial port and the parallel port of the guest without the virtualbox guest tools installed.
Unfortunately it doesn't offer a text console interface but the serial port can be connected to a local pipe file and that can probably be worked with just as well.
This may not be exactly what you need:
I have done something similar with a Ubuntu-based install. We used preseeding (Debian's form of kickstart), to answer all the questions during the install - providing the preseed file and the installer via tftp.
In addition to the official Ubuntu mirror we added the apt-server with our own packages in the preseed file. We put a .deb version of vmware-tools on the apt-server and added it to the packages to be installed.
The .deb of vmware tools just contained the .tar.gz and a postinstall script that would extract it to /tmp and run the vmware install script (which has a switch to be run unnattended, so it does not ask any questions).
So after the reboot vmware-tools were up and running and we could use vix to script the rest (which was not very reliable).
If you should encounter problems with running vmware-config.pl during boot, you could make a custom package that just extracts the tools and an init script that installs them on first boot, disables itself and reboots.
Maybe you can use this strategy (replacing apt by yum, preseed by kickstart and tftp by a remastered iso). If you really need to test that your users choose a keyboard in the installer (which is not very different from kickstart) this would obviously not work for you..