Apache Zeppelin + EMR(Spark) Cluster for opening the firewall - amazon-emr

I tried to use apache zeppelin with EMR(Spark) Cluster. I get some requirements for using apache zeppelin + EMR cluster with opening firewall. In the workplace, there is the static ip which is blocked by the firewall. As you know,
EMR clusters should change their IP and DNS name every time when they create with aws cli command. So do you know how to connect apache zeppelin server(EC2 instance) with the EMR cluster using the fixed IP?
Thanks in advance.

I don't understand your question fully. Let me try to answer this part of question: "So do you know how to connect apache zeppelin server(EC2 instance) with the EMR cluster using the fixed IP?"
This should be possible by attaching an elastic ip with EMR clusters' master using: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-cli-commands.html#elastic-ip.
You can also try qubole's managed clusters and support for spark+zeppelin. Qubole takes care of this by providing you a fixed end point to access your zeppelin notebooks.
Disclaimer: I work for Qubole

Finally I succeed to solve this problem or requirement using socat.
socat TCP-LISTEN:8080,fork TCP:$EMR_CLUSTER_NAME:8080
socat TCP-LISTEN:8081,fork TCP:$EMR_CLUSTER_NAME:8081
Also check the script(install-apache-zeppelin-on-amazon-emr.sh) which I revised some codes and used socat instead of using ssh tunneling.

Related

Cannot ssh to google cloud instance

I'm newbie for GCP and I need your help which this is the step I had made.
(1) I setup google cloud firewall rules to allow ssh on port 22 and I can ssh to my instance, CentOS7, correctly.
(2) When I connect to my instance, I run some firewall script and after that I cannot ssh to my instance anymore. It seem that script block ssh port even I enable it in the VPC Network > Firewall rules.
(3) Now I cannot connect to my instance including Open in browser window in the SSH menu on gcp console.
Is there any solution to connect my instance? Please help.
Thank in advance.
Bom
You probably change block ssh port by changing firewall configuration inside VM.
So you can consider 2 options :
1) Recreate VM if no sensitive data, or not too much work spent for the existing setup.
2) Detach Boot disk and reuse it on another instance, to change the configuration files of firewal.
check Official Docs - Use your disk on a new instance for that:
gcloud compute instances delete $PROB_INSTANCE
--keep-disks=boot
gcloud compute instances create new-instance
--disk name=$BOOT_DISK,boot=yes,auto-delete=no
gcloud compute ssh new-instance
Hope it will help you.

I'm using a kuberctl(on my computer) to use the minikuber(on the other server), but how to start it?

The OS: Ubuntu 16.04;
On my computer: need a kuberctl to use the minikube,
On the server: a minikube on it and running already,
The net: both my computer and server are in one net, route is OK,
Question: I have downloaded a kuberctl binary tag.gz already, how to start and config my kuberctl?
I've unzipped the kuberctl binary tag.gz, but I don't know what to do next...
In fact, no code by now.
I hope to config the kuberctl and start it; the documentation seems help nothing.
Please follow instructions described in the documentation. You need to specify connection properties for your kubectl binary using a configuration file. To do that you'll need to know the Minikube configuration on your server.
Make sure that the minikube cluster is accepting connections from external addresses and there are no firewall rules blocking the connection.
After that, you should be able to use kubectl on your local computer to control that minikube cluster.
Also, it's kubectl and not kuberctl.

Mesos Failed to connect error to IP:5050

I am new to Mesos and just finished setting up mesos and along with zookeeper on my test server.
Unfortunately I keep getting this error message on my mesos console indicating i am unable to connect to mesos on port 5050 and can't seem to figure out why.
I have included the error in the screen shot below
The mesos log files doesn't point to why the error is showing either.
I resolved the problem by this:
./bin/mesos-master.sh --ip=x.x.x.x --work_dir=/var/lib/mesos --hostname=x.x.x.x
We can avoid this problem by starting mesos-master with following option:
--ip=xx.xx.xx.xx --hostname_lookup=false
I have resolved this problem. Open the web page in Chrome, and open the developer tool, you will see the chrome is accessing the web site with domain, in my case the domain name is "mesosphere", as there is no mesosphere in dns, so the accessing was failed.
I solved the problem by adding the mesosphere in the hosts file, C:/windows/system32/etc/hosts/
If you use the domain name for the Mesos cluster you must set the domain name in windows hosts.
There can be multiple issues here.
Is your mesos-master running and healthy ?
Has leader election process completed, if all is good.
Check if you are able to do
ping leader.mesos
If above ping doesn't work, that means leader has not been elected. First fix that.
I had this problem also. Luckily, I have a running mesos server also. So, I can compare the different between my demo and the running mesos server. I captured the packets between client and server in my demo. I found the explorer didn`t resend fresh request, only some keepalive packets.
but, when I catch the packets in the running mesos server, I found the explorer send get request frequently. like the image
I think, if you run some task or add some agent, maybe it will activate the explore to send request frequently. Then the "Failed to connect" will disappeared.
I was having the same issues and what fixed it for me was the zookeeper configuration. In my case I was using the EC2 public IP Address rather than the private one. Once I changed the /etc/mesos/zk file to zk://<private IP>:2181/mesos I was able to connect without the constant error messages. In other words, zookeeper was reporting to be running in one IP and mesos-master was trying to connect using a different IP.
My configuration was correct as suggested. But failed to start mesos-master service. But There is alternative way to start mesos-master node with exact same configuration. Commands to start mesos-master
$ cd /usr/sbin [or mesos_installation directory/bin]
$sudo ./mesos-master --work_dir=/var/lib/mesos --log_dir=/home/rajeev/logs/mesos/
Its start mesos-master service successfully for me.

Can't connect apache running on Amazon EC2

Today I got one instance in Amazon EC2 with Ubuntu 14. Through SSH( A Java SSH Client directly from my browser (Java required) ) I am able to login and able to install LAMP server.
I also include Inbound rules in Security group.
But when I am trying to load apache from browser I got "This site is configured to require an SSL (https) connection." this error.
While trying with https then got unable to connect.
I didn't under the issue.
I have added ICMP in the security group and ping is working from my local.
Could you please help me to resolve this.
Thanks in advance.
I selected Ubuntu as operating system but I took another instance with Amazon Linux and it is working now.

Amazon EC2 ssh timeout

I am new to AWS services. I created an EC2 instance of Ubuntu and I was able to connect to it. I was able to run TeamCity and Rails server on the instance and things were fine. I created an image of the instance and now I can not ssh back.
Help is much appreciated. Thanks!
which country are you from? i know in china, the instances of ec2 cannot be connected unless you use a VPN tool.