EMR uses Yarn but there is an option to run on AKS with k8s as resource manager. What about serverless EMR, it seems that the application looks like a multimaster k8s deployment so I would assume that k8s is the resource manager, any pointers?
thank,
Mike
architectural question
Related
How to create an IAM role to run workloads on Amazon EMR on EKS?
The official documentation remains very vague on this point in particular
This is multistep process. We have EKS cluster on which we run EMR jobs. So we need to setup IAM + EKS role mapping configurations. AWS Documentation is detailed which creates confusion sometimes.
Following are high level steps which might help you to refer AWS documentation again and understand.
Note: These steps assumes you already have created EKS.
Create Namespace to run your spark jobs.
Create RBAC Role and Role binding for your cluster configuration. (This is EKS level role based access mechanism.)
Edit aws-auth to update roleARN for AWSServiceRoleForAmazonEMRContainers. (EKS Authorisation)
Create EMR virtual cluster and assigned it to EKS namespace created above.
Create trust policy to access EKS cluster for EMR containers.
Create Job execution role and associate above policy to it.
Submit your emr on eks job.
On the Ignite website, I see that in Amazon EKS, Microsoft Azure Kubernetes Service Deployment, and Google Kubernetes Engine Deployment, deploy on each of the three platforms ignite.If I am on my own deployed K8S, can I deploy?Is it the same as deploying the Ignite on three platforms?
Sure, just skip the initial EKS/Azure initialization steps since you don't need them and move directly to the K8s configuration.
Alternatively, you might try Apache Ignite and GridGain k8s operator that simplifies the deployment.
I was planning to use Jenkins(CI) ----> Spinnaker(CD) integration for AWS EKS.
Does Spinnaker support multi-cluster deployments?
For example:
I will have 4 clusters in different accounts
and I want to have 1 Spinnaker deployed to one of the clusters and manage other 3 as well.
is it possible to do?
Yes is possible. The suggested way is to have Spinnaker Running in an AWS Account called Spinnaker or CD and in a specific namespace called spin.
A great guide to follow for Spinnaker in EKS is Continuous Delivery using Spinnaker on Amazon EKS
I want deploy a s3 compatible blob storage in my Kubernetes Cluster. I already use GlusterFS for volumes like mongodb, and I tried to set up minio with the helm chart https://github.com/helm/charts/tree/master/stable/minio. I just realize I can't scale up minio easily because of erasure code.
So I have some questions about blob storage solutions :
Is GlusterFS blob storage service stable and reliable (https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/gluster-s3-storage-template) ?
Do I must use OpenShift to deploy GlusterFS blob storage as I read in the web ? I think no because I can see simple Kubernetes manifests in the GlusterFS repo like this one : https://github.com/gluster/gluster-kubernetes/blob/master/deploy/kube-templates/gluster-s3-template.yaml.
Is it easy to use Minio federation in Kubernetes ? Is it easily scalable with a "helm upgrade --set replicas=X" or do I need manually upgrade minio configuration ?
As you can see, I feel lost with this s3 storage. So if you have more information/solutions, do not hesitate.
Thanks in advance !
About reliability you should read more about user experience like:
An end user review of GlusterFS
Community Survey Feedback, 2019
Why openshift with glusterFS:
For standalone Red Hat Gluster Storage, there is no component installation required to use it with OpenShift Container Platform. OpenShift Container Platform comes with a built-in GlusterFS volume driver, allowing it to make use of existing volumes on existing clusters but Red Hat Gluster Storage is a commercial storage software product, based on Gluster.
How to deploy it in AWS
For minio please follow official docs:
ConfigMap allows injecting containers with configuration data even while a Helm release is deployed.
To update your MinIO server configuration while it is deployed in a release, you need to
Check all the configurable values in the MinIO chart using helm inspect values stable/minio.
Override the minio_server_config settings in a YAML formatted file, and then pass that file like this helm upgrade -f config.yaml stable/minio.
Restart the MinIO server(s) for the changes to take effect
I didn't try but, but as per documentation:
For federation I can see additional environment variables in the values.yaml.
In addition you should Run MinIO in federated mode Federation Quickstart Guide
Here you can find differences between google and amazon s3 sotrage
or Cloud Storage interoperability from gcloud perspective.
Hope this help.
I was planning on installing Istio on my new AKS cluster. However, in the prerequisites for Istio, it is mentioned that K8S cluster should have RBAC enabled. However, I read that AKS ( preview ) doesn't have it enabled. Is this true? Is there an option for me to try Istio on AKS.
AKS is GA and looks like RBAC is available now,
https://azure.microsoft.com/en-us/blog/azure-kubernetes-service-aks-ga-new-regions-new-features-new-productivity/
In fact RBAC is not available in Azure AKS currently. According to this GitHub issue it is on the roadmap for Q1 2018.
In Azure you can use ACS which is an older version of AKS, but with control over Kubernetes master or acs-engine where you have full control over Kubernetes cluster.
AKS now enables RBAC by default.
There are also docs on how to install Istio:
https://learn.microsoft.com/en-us/azure/aks/istio-install
We are doing POC for service mesh on our AKS cluster using istio. I have found a very good guide to install istio with all its components on AKS cluster and it does not require any RBAC on AKS. Infact this guide is cloud agnostic. I am not sure if it production graded ready but working like a charm till now. Just apply first 3 files and 4th one optional. The name might be little confusing for you. But its working on AKS very well. Hope that worsk for you.
Istio Installation Files
kubectl apply -f 1-istio-init.yaml 2-istio-minikube.yaml 3-kiali-secret.yaml