What is the simplest way to add livy interpreter to Zeppelin running on EMR cluster. What would be the right step to add to get it?
Related
I have been trying several solutions with custom jars from redislab and also with --packages in spark submit emr and still no suceess is there any simple way in emr to connect to elasticache ?
I'm execution a process in a cluster emr with steps that executes a command s3-dist-cp to copy files from my s3 to EMR.
I'm having an error Input path does not exist. But the ip of machine in log does not is part of cluster.
Did someone already face this problem?
Does EMR 5.4.0 supports HA for Resource Manager, Namenode and Hive? if not any road map for the same?
i am not able to get it from the EMR documentation site
https://docs.aws.amazon.com/emr/latest/ReleaseGuide
please suggest if you find any useful document
As of Feb 2018 AWS EMR has a single point of failure at Master Node.
EMR FAQ
Look at :
Q: If the master node in a cluster goes down, can Amazon EMR recover it?
If HA is necessary requirement then you would want to consider either Cloud Offerings from Cloudera/Hortonworks/Mapr or Custom Installations on AWS EC2s.
To help people coming to this post for EMR HA queries.
Finally, as of writing this on may 2019, HA is made GA on AWS EMR. More information in the AWS post below.
https://aws.amazon.com/about-aws/whats-new/2019/04/amazon-emr-announces-support-for-multiple-master-nodes-to-enable-high-availability-for-EMR-applications/
I have listed few queries related to ambari as follows:
Can I configure more than one hadoop cluster via UI of ambari ?
( Note : I am using Ambari 1.6.1 for my hadoop cluster deployment purpose and I am aware that this can be done via Ambari API, but not able to find via ambari portal)
We can check the status of services on each node by “jps” command, if we have configured hadoop cluster w/o ambari.
Is there any way similar to “jps” to check from back end if the setup for hadoop cluster was successful from the backend ?
( Note : I can see that services are showing UP on the ambari portal )
Help Appreciated !!
Please let me know if any additional information is required.
Thanks
The ambari UI is served up by the ambari server for a specific cluster. To configure another cluster you need to point your browser to the URL for that other cluster's ambari server. So you can't see the configuration for multiple servers on the same web page, but you can set browser bookmarks to jump from configuration to configuration.
I'm using Amazon Elastic MapReduce Ruby (http://aws.amazon.com/developertools/2264) to run my hive job. Is there a way to know when the job is done? Right now all I could think of is the keep running emrclient with "--list --active" but I'm hoping there is a better way to do this.
Thank you
You may also get to know this from the aws console's EMR section.
If your concern is to terminate the cluster once your job is done then while launching the cluster don not use the option --stay-alive. Or alternatively, you can have a script which would poll for the current status of the running cluster and terminate it once it gets to 'waiting' state.
I do not think there is another way.