I have 2 projects, each one should use a single cluster, but I don't know how to submit job to different YARN cluster?
I have tried to use the
-m,--jobmanager
Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration.
It does not work.
Related
I am a newbie of GKE. I created a GKE cluster with very simple setup. It only has on gpu node and all other stuff was default. After the cluster is up, I was able to list the nodes and ssh into the nodes. But I have two questions here.
I tried to install nvidia driver using the command:
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
It output that:
kubectl apply --filename https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
daemonset.apps/nvidia-driver-installer configured
But 'nvidia-smi' cannot be found at all. Should I do something else to make it work?
On the worker node, there wasn't the .kube directory and the file 'config'. I had to copy it from the master node to the worker node to make things work. And the config file on the master node automatically updates so I have to copy again and again. Did I miss some steps in the creation of the cluster or how to resolve this problem?
I appreciate someone can shed some light on this. It drove me crazy after working on it for several days.
Tons of thanks.
Alex.
For the DaemonSet to work, you need to have a tag on your worker Node as cloud.google.com/gke-accelerator (see this line). The DaemonSet checks for this tag on a node before scheduling any pods for installing the driver. I'm guessing a default node pool you create did not have this tag on it. You can find more details on this on the GKE docs here.
The worker nodes, by design are just that worker nodes. They do not need privileged access to the Kubernetes API so they don't need any kubeconfig files. The communication between worker nodes and the API is strictly controlled through the kubelet binary running on the node. Therefore, you will never find kubeconfig files on a worker node. Also, you should never put them on the worker node either, since if a node gets compromised, the keys in that file can be used to damage the API Server. Instead, you should make it a habit to either use the master nodes for kubectl commands, or better yet, have the kubeconfig on your local machine, and keep it safe, and issue commands remotely to your cluster.
After all, all you need is access to an API endpoint for your Kubernetes API server, and it shouldn't matter where you access it from, as long as the endpoint is reachable. So, there is no need whatsoever to have kubeconfig on the worker nodes :)
Can Redis run in cluster mode enabled but with single instance. It seems we get cluster status as fail when we try to deploy with single node because no slots are added to it.
https://serverfault.com/a/815813/510599
I understand we can manually add SLOTS after deployment. But I wonder if we can modify Redis source code to make this change.
I have several machines on my intranet. If I switch on ignite on two of them, they automatically discover each other and become part of a single cluster. If I start ignite on a third machine, it automatically connects to the cluster.
How can I prevent this.
Basically, I want to run two clusters of Ignite on a single network. I have two testing environments, I want separate Ignites for both these environments.
I suppose that you're using TcpDiscoveryMulticastIpFinder in TcpDiscoverySpi configuration.
It's possible to archive network isolation, but you should use TcpDiscoveryVmIpFinder instead of TcpDiscoveryMulticastIpFinder. The example of configuration could be found here https://apacheignite.readme.io/docs/tcpip-discovery#section-static-ip-finder.
Is there any way of running a service (single instance) on each deployed agent node? I need that because each agent needs to mount a storage from S3 using s3fs
The name of the feature you're looking for is "daemon tasks", but unfortunately, it's still in the planning phase for Mesos itself.
Due to the fact that schedulers don't know the entire state of the cluster, Mesos needs to add a feature to enable this functionality. Once in Mesos it can be integrated with DC/OS.
The primary workaround is to use Marathon to deploy an app with the UNIQUE constraint ("constraints": [["hostname", "UNIQUE"]]) and set the app instances to the number of agent nodes. Unfortunately this means you have to adjust the instances number when you add new nodes.
Am wondering if there is a way two logstash processes with separate configurations can be run on a single Docker container.
My setup has a Logstash process using file as input sending events to Redis and from there to second Logstash process and over to custom http process. So, Logstash --> Redis --> Logstash --> Http. Was hoping to keep the two Logstash instances and Redis on the same Docker container. Am still new to Docker & Would highly appreciate any inputs / feedback on the same.
This would be more complicated than it needs to be. It is much simpler in the Docker world to run three containers to do three things than to run one container that does them all. It is possible though-
You need to run an init process in your container to control multiple processes, and launch that as your container's entry point. The init will have to know how to launch the processes you are interested in, both logstash and the redis. Basimage/phusion provides an image with a good init system, but the launch scripts are based on runit and can be hard to pick up.
If you wanted to only run a single process, you can use a docker-compose file to launch all three processes and link them together.