I am trying to transfer file from cloud bucket to VM instance.
I connected to VM instance by SSH, And used gsutil code like this :
gsutil cp gs://[bucket name]/[file name] /home
and then error was occur
Copying gs://huji/final-project-deep-learning-19-tf.tar...
OSError: Permission denied. GiB]
So I tried this command :
sudo gsutil cp gs://[bucket name]/[file name] /home
And then, downloading completed.
Copying gs://huji/final-project-deep-learning-19-tf.tar...
| [1 files][ 1.7 GiB/ 1.7 GiB] 76.2 MiB/s
but I cannot find any files and the disk volume hasn't changed.
Can anybody explain why this happens and how to copy files from cloud? thanks.
Create a service account link:
gcloud iam service-accounts create transfer --description "transfer" --display-name "transfer"
Set up a new instance to run as a service account:
gcloud compute instances create instance --service-account transfer#my-project.iam.gserviceaccount.com --scopes https://www.googleapis.com/auth/devstorage.full_control
For example, the scope for full access to Cloud Storage is
https://www.googleapis.com/auth/devstorage.full_control. The alias for
this scope is storage-full
SSH to the VM instance:
gcloud beta compute --project "my-project" ssh --zone "us-central1-a" "instance"
Copy the files :
gsutil cp gs://bucket/file.csv .
List the files:
ls
file.csv
Related
Below are steps i preformed to use custom AMI EKS managed node group.
bootstrap_user_data file has been created and its converted to base64 format as per the standard.
#!/bin/bash
set -ex
B64_CLUSTER_CA= <My eks cluster Certificate authority value>
API_SERVER_URL= <My EKS cluster API server URl>
/etc/eks/bootstrap.sh <cluster-name> --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL
cat bootstrap_user_data | base64
Launch template created via custom-configuration.json file with below data
cat config_custom_ami.json
{
"LaunchTemplateData": {
"EbsOptimized": false,
"ImageId": "ami-0e00c1f097aff7fe8",
"InstanceType": "t3.small",
"UserData": "bootstrap_user_data",
"SecurityGroupIds": [
"sg-0e9b58499f42bcd4b"
]
}
}
Security group has been selected EKS cluster security group it was created automatically while creating EKS cluster first time.
creating launch template using eksctl command
aws ec2 create-launch-template --region eu-central-1 --launch-template-name my-template-name --version-description "first version " --cli-input-json file://custom.config.json
creating node group using eksctl command
aws eks create-nodegroup --region eu-central-1 --cluster-name my-cluster --nodegroup-name my-node-group --subnets subnet-<subnet1> subnet-<subnet2> --node-role 'arn:aws:iam::123456789:role/EKSNODEGROUP' --launch-template name=my-template-name
After executing node group creation command it was taking 20 min to create node group at the same time desired VM is created as part of auto scaling group but nodes group not able to join to the cluster after 20 min.
Connect to your Amazon EKS worker node instance with SSH and check kubelet agent logs
ssh -i my.key ec2-user#1.2.3.4
sudo -i
cd /etc/eks/bootstrap.sh
-bash: cd: /etc/eks: No such file or directory
could you please some one help why my bootstrap.sh file not exists inside the /etc/eks location in other hand in AWS console launch template - Advanced tab - i can able to see my user data in decoded format.
I am having no trouble sshing into a Google Cloud compute engine VM, but am unable to ssh into the master node of a Google Cloud Dataproc cluster.
Specifically,
gcloud compute ssh my-vm
works just fine, while
gcloud compute ssh mycluster-m
fails with error message:
admin#IP.ADDRESS: Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
The compute engine VM and the Dataproc cluster are in the same project. I understand from the error message it is something related to the ssh keys, but I am not sure how to fix it - I checked the ssh keys in the project via cloud console, and it is correct, and tried the usual gcloud auth login to reset gcloud project login details.
Any hints on how to fix this?
Edits: I am trying to ssh from my machine, not the cloud console- that's a good point, I will try that and see if that is possible. But in the end I want to use this to connect to a Jupyter notebook from my local computer, so that does not solve the issue of being unable to SSH from my machine to the VM.
Concerning the command to create the Dataproc cluster, I use tools from the hail dataproc python library, but these are basically just convenience shells for the gcloud compute commands, and this is what is failing. But the command I used to create the Dataproc cluster was:
gcloud beta dataproc clusters create \
test \
--image-version=1.4-debian9 \
--properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=41g \
--initialization-actions=gs://hail-common/hailctl/dataproc/0.2.53/init_notebook.py \
--metadata=^|||^WHEEL=gs://hail-common/hailctl/dataproc/0.2.53/hail-0.2.53-py3-none-any.whl|||PKGS=aiohttp>=3.6,<3.7|aiohttp_session>=2.7,<2.8|asyncinit>=0.2.4,<0.3|bokeh>1.1,<1.3|decorator<5|dill>=0.3.1.1,<0.4|gcsfs==0.2.1|humanize==1.0.0|hurry.filesize==0.9|nest_asyncio|numpy<2|pandas>0.24,<0.26|parsimonious<0.9|PyJWT|python-json-logger==0.1.11|requests>=2.21.0,<2.21.1|scipy>1.2,<1.4|tabulate==0.8.3|tqdm==4.42.1|google-cloud-storage==1.25.* \
--master-machine-type=n1-highmem-8 \
--master-boot-disk-size=100GB \
--num-master-local-ssds=0 \
--num-preemptible-workers=0 \
--num-worker-local-ssds=0 \
--num-workers=2 \
--preemptible-worker-boot-disk-size=40GB \
--worker-boot-disk-size=40GB \
--worker-machine-type=n1-standard-8 \
--initialization-action-timeout=20m \
--labels=creator=my_name \
--max-idle=10m
Turns out the problem is that the cluster creates a new account called my_username on the cluster master VM, but I am logged into my laptop as a user called 'admin'. So there is a mismatch between account name and key at the destination, so the login fails.
Can be fixed by adding username to the gcloud command:
gcloud compute ssh my_username#mycluster-m
Though I still don't really understand why the ssh keys are different for the dataproc VM and a compute engine VM, I'd be happy if someone can enlighten me.
I installed spinnaker on my AWS EC2, login into the dashboard in the first time but immediately after I logout and login again using the same base URL i am being directed to a different person github account, what might have happened, does it mean my account is hacked or what, somebody advise please.
Being directed to the link attached below, instead of the ip address taking me to the spinnaker dashboard and yet I am using the correct base address
These are the instructions i follow for Minnaker on EC2 (ap-southeast-2)
Pre-requisites
Obtain an AWS Elastic IP
From AWS EC2 console choose a Region preferably ap-southeast-2 and
launch an EC2 instance with 16 GB memory, 4 cpu min and 60 GB disk.
An initial deployment can be performed using instance= m4.xlarge
Attach the AWS Elastic IP to the Spinnaker Instance
Access the instance through SSH
Get minnaker
curl -LO https://github.com/armory/minnaker/releases/latest/download/minnaker.tgz
Untar
tar -xzvf minnaker.tgz
Go to minnaker directory
cd minnaker
Use the Public IP value from The Elastic IP as the $PUBLIC_IP
Obtain Private IP of the instance hostname -I and add them to local environment variables $PRIVATE_IP
export PRIVATE_IP=$(hostname -I)
export PUBLIC_IP=AWS_ELASTIC_IP
Execute the command below to install Open Source Spinnaker
./scripts/install.sh -o -P $PRIVATE_IP
Validate installation
UI
Validate installation going to generated URL https://PUBLIC_IP
Use user admin and get the password at etc/spinnaker/.hal/.secret/spinnaker_password
The UI should load
Kubernetes Deployment
Minnaker is deployed inside an EC2 as a lightweight Kubernetes K3S cluster
Run kubectl version
Get info from cluster kubectl cluster-info
Tweak bash completion and enable a simple alias.
kubectl completion bash
kubectl completion bash
echo 'source <(kubectl completion bash)' >>~/.bashrc
kubectl completion bash >/etc/bash_completion.d/kubectl
echo 'alias k=kubectl' >>~/.bashrc
`echo 'complete -F __start_kubectl k' >>~/.bashrc
Validate Spinnaker is running
k -n spinnaker get pods -o wide
Halyard Config
Validate a default halyard config is been set up
sudo chmod 755 /usr/local/bin/hal
#!/bin/bash
set -x
HALYARD=$(kubectl -n spinnaker get pod -l app=halyard -oname | cut -d'/' -f 2)
k -n spinnaker exec -it ${HAYLYARD} -- hal $# config
Minnaker repo
Clone the repository
Go to Scripts directory cd minnaker/scripts
Add permissions to the installation script chmod 775 all.sh
git clone https://github.com/armory/minnaker
References
armory/minnaker
I have a single master, multinode kubernetes going. It works great. However I want to allow kubectl commands to be run from outside the master server. How do I run kubectl get node from my laptop for example?
If I install kubectl on my laptop I get the following error:
error: client-key-data or client-key must be specified for kubernetes-admin to use the clientCert authentication method
How do I go about this. I have read through the kubernetes authorisation documentation but I must say it's a bit greek to me. I am running version 1.10.2.
Thank you.
To extend #sfgroups answer:
Configurations of all Kubernetes clusters you are managing
are stored in $HOME/.kube/config file. If you have that file on the master node,
the easy way is to copy it to $HOME/.kube/config file on a local machine.
You can choose other places, and then specify the location by environment value KUBECONFIG:
export KUBECONFIG=/etc/kubernetes/config
or use --kubeconfig command line parameter instead.
Cloud providers often give you a possibility to download config to local machine from the
web interface or by the cloud management command.
For GCP:
gcloud container clusters get-credentials NAME [--region=REGION | --zone=ZONE, -z ZONE] [GCLOUD_WIDE_FLAG …]
For Azure:
az login -u yourazureaccount -p yourpassword
az acs kubernetes get-credentials --resource-group=<cluster-resource-group> --name=<cluster-name>
If the cluster was created using Kops utility, you could get the config file by:
kops export kubeconfig ${CLUSTER_NAME}
From your master copy /root/.kube directory to your laptop C:\Users\.kube location.
kubectl will pickup the certificate from config file automatically.
Say I have a website that return me JSON data when I send a GET request using curl. I want to re-direct the output of curl to AWS S3. A new file should be created on S3 for it.
Currently I am able to redirect the output to store it locally.
curl -s -X GET 'http://website_that_returns_json.com' > folder_to_save/$(date +"%Y-%m-%d_%H-%M.json")
I have AWS CLI and s3cmd installed. How would I redirect the output of create to create a new file on AWS S3 ?
Assume :
AWS S3 access key and secret key are already set.
Location to store file : mybucket/$(date +"%Y-%m-%d_%H-%M.json"
The AWS Command-Line Interface (CLI) has the ability to stream data to/from Amazon S3:
The following cp command uploads a local file stream from standard input to a specified bucket and key:
aws s3 cp - s3://mybucket/stream.txt
So, you could use:
curl xxx | aws s3 cp - s3://mybucket/object.txt
However, it's probably safer to save the file locally and then copy it to Amazon S3.
In case you'd like to run the command on the remote, use aws ssm send-command.
Then to redirect the output of that command to S3, you can use --output-s3-bucket-name parameter.
Here is Bash script to run PowerShell script on the remote and upload it into S3 bucket:
instanceId="i-xyz"
bucketName="bucket_to_save"
bucketDir="folder_to_save"
command="Invoke-WebRequest -UseBasicParsing -Uri http://example.com).Content"
cmdId=$(aws ssm send-command --instance-ids "$instanceId" --document-name "AWS-RunPowerShellScript" --query "Command.CommandId" --output text --output-s3-bucket-name "$bucketName" --output-s3-key-prefix "$bucketDir" --parameters commands="'${command}'")
while [ "$(aws ssm list-command-invocations --command-id "$cmdId" --query "CommandInvocations[].Status" --output text)" == "InProgress" ]; do sleep 1; done
outputPath=$(aws ssm list-command-invocations --command-id "$cmdId" --details --query "CommandInvocations[].CommandPlugins[].OutputS3KeyPrefix" --output text)
echo "Command output uploaded at: s3://${bucketName}/${outputPath}"
aws s3 ls "s3://${bucketName}/${outputPath}"
To output the uploaded S3 files, run:
aws s3 ls s3://${bucketName}/${outputPath}/stderr.txt && aws s3 cp --quiet s3://${bucketName}/${outputPath}/stderr.txt /dev/stderr
aws s3 cp --quiet s3://${bucketName}/${outputPath}/stdout.txt /dev/stdout