How to create a service networking creation from a gcp project to "default" network - sql

I wanted to use the gcloud cli to create an sql instance that is accessible on the default network. So I tried this:
gcloud beta sql instances create instance1 \
--network projects/peak-freedom-xxxxx/global/networks/default
And I get the error
ERROR: (gcloud.beta.sql.instances.create) [INTERNAL_ERROR] Failed to create subnetwork.
Please create Service Networking connection with service 'servicenetworking.googleapis.com'
from consumer project '56xxxxxxxxx' network 'default' again.
When you go to the console to create it, you can check Private IP you can see this:
And there's an "Allocate and connect" button. So I'm guessing that's what I need to do. But I can't figure out how to do that with the gcloud cli.
Can anyone help?
EDIT 1:
I've tried setting the --network to https://www.googleapis.com/compute/alpha/projects/testing-project-xxx/global/networks/default
Which resulted in
ERROR: (gcloud.beta.sql.instances.create) [INTERNAL_ERROR] Failed to
create subnetwork. Set Service Networking service account as
servicenetworking.serviceAgent role on consumer project
Then I tried recreating a completely new project and enabling the Service Networking API like so:
gcloud --project testing-project-xxx \
services enable \
servicenetworking.googleapis.com
And then creating the DB resulted in the same error. So I tried to manually add the servicenetworking.serviceAgent role and ran:
gcloud projects add-iam-policy-binding testing-project-xxx \
--member=serviceAccount:service-PROJECTNUMBER#service-networking.iam.gserviceaccount.com \
--role=roles/servicenetworking.serviceAgent
This succeeded with
Updated IAM policy for project [testing-project-xxx].
bindings:
- members:
- user:email#gmail.com
role: roles/owner
- members:
- serviceAccount:service-OJECTNUMBERRP#service-networking.iam.gserviceaccount.com
role: roles/servicenetworking.serviceAgent
etag: XxXxXX37XX0=
version: 1
But creating the DB failed with the same error. For reference, this is the command line I'm using to create the DB:
gcloud --project testing-project-xxx \
beta sql instances create instanceName \
--network=https://www.googleapis.com/compute/alpha/projects/testing-project-xxx/global/networks/default \
--database-version POSTGRES_11 \
--zone europe-north1-a \
--tier db-g1-small

the network name of form "projects/peak-freedom-xxxxx/global/networks/default" is for creating SQL instances under shared VPC network. if you want to create an instance in a normal VPC network you should use:
gcloud --project=[PROJECT_ID] beta sql instances create [INSTANCE_ID]
--network=[VPC_NETWORK_NAME]
--no-assign-ip
where [VPC_NETWORK_NAME] is of the form https://www.googleapis.com/compute/alpha/projects/[PROJECT_ID]/global/networks/[VPC_NETWORK_NAME]
for more information check here.
Note: you need to configure private service access for this and it's one time action only. follow instructions here to do so.

Related

gcloud compute ssh requires password even after using json key file for authentication

I am trying to authenticate gcloud using json key and even doing everything as per docs it requires for password when I run gcloud compute ssh root#production
Here is snapshot of steps I performed.
1. Authorizing access to Google Cloud Platform with a service account
tahir#NX00510:~/www/helloworld$ gcloud auth activate-service-account 1055703200677-compute#developer.gserviceaccount.com --key-file=gcloud_key.json
Activated service account credentials for: [1055703200677-compute#developer.gserviceaccount.com]
2. Initializing the gcloud
tahir#NX00510:~/www/helloworld$ gcloud init
Welcome! This command will take you through the configuration of gcloud.
Settings from your current configuration [default] are:
compute:
region: us-central1
zone: us-central1-b
core:
account: 1055703200677-compute#developer.gserviceaccount.com
disable_usage_reporting: 'True'
project: concise-hello-122320
Pick configuration to use:
[1] Re-initialize this configuration [default] with new settings
[2] Create a new configuration
Please enter your numeric choice: 1
Your current configuration has been set to: [default]
You can skip diagnostics next time by using the following flag:
gcloud init --skip-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).
Choose the account you would like to use to perform operations for
this configuration:
[1] 1055703200677-compute#developer.gserviceaccount.com
[2] Log in with a new account
Please enter your numeric choice: 1
You are logged in as: [1055703200677-compute#developer.gserviceaccount.com].
API [cloudresourcemanager.googleapis.com] not enabled on project
[1055703200677]. Would you like to enable and retry (this will take a
few minutes)? (y/N)? N
WARNING: Listing available projects failed: PERMISSION_DENIED: Cloud Resource Manager API has not been used in project 1055703200677 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=1055703200677 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
- '#type': type.googleapis.com/google.rpc.Help
links:
- description: Google developers console API activation
url: https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=1055703200677
Enter project id you would like to use: concise-hello-122320
Your current project has been set to: [concise-hello-122320].
Do you want to configure a default Compute Region and Zone? (Y/n)? n
Your Google Cloud SDK is configured and ready to use!
* Commands that require authentication will use 1055703200677-compute#developer.gserviceaccount.com by default
* Commands will reference project `concise-hello-122320` by default
Run `gcloud help config` to learn how to change individual settings
This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.
Some things to try next:
* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting
3. SSHing to gcloud
tahir#NX00510:~/www/helloworld$ gcloud compute ssh root#production
No zone specified. Using zone [us-central1-b] for instance: [production].
root#compute.1487950061407628967's password:
I don't know which password should I enter here, also I believe it should not ask for password in the first place because I have used json key file for authentication.
Could you guys please help me out to fix this.
Thanks !

How to read data from PostgreSQL database (JDBC) and write to BigQuery?

I have my database in one of my project in GCP with shared VPC.
From a different project, I started google data flow, and tried to connect the database, but always get 'The connection attempt failed.', since it couldn't find the hostname that I provided in my application.yml.
If I create a VM instance in this Big Query project (with shared VPC network), and install PostgreSQL client, I can connect to that database.
I checked this question, but I don't see the answer for connecting to database.
I also, tried by running maven compile with subnetworks parameter either complete or short form URL, but still no luck.
Below is the command that I ran from cloudshell:
mvn compile exec:java -Dexec.mainClass=com.myurl.spring.beam.pipeline.SpringBeamPipelineLauncher \
-Dexec.args="--project=test-big-query \
--stagingLocation=gs://test-big-query/test/ \
--runner=DataflowRunner \
--jobName=dataflow-dryrun \
--springContextClass=com.myurl.bi.gcdf.sqlloader.config.PipelineConfig \
--bootstrapOptionClass=com.myurl.spring.beam.pipeline.options.BootstrapOptions \
--entryPoint=com.myurl.bi.gcdf.sqlloader.SQLLoaderEntryPoint \
--subnetwork=https://www.googleapis.com/compute/v1/projects/NAMEOFVPCHOSTPROJECT/regions/us-central1/subnetworks/NAMEOFVPCNETWORK"
Can you confirm that you are specifying the name of the subnet and not the name of the vpc?
--subnetwork=https://www.googleapis.com/compute/v1/projects/NAMEOFVPCHOSTPROJECT/regions/us-central1/subnetworks/MYSUBNET"
If you can reach your pgsql instance using a client in MYSUBNET then specifying the above parameter should work.

GKE clusterrolebinding for cluster-admin fails with permission error

I've just created a new cluster using Google Container Engine running Kubernetes 1.7.5, with the new RBAC permissions enabled. I've run into a problem allocating permissions for some of my services which lead me to the following:
The docs for using container engine with RBAC state that the user must be granted the ability to create authorization roles by running the following command:
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>]
However, this fails due to lack of permissions (which I would assume are the very same permissions which we are attempting to grant by running the above command).
Error from server (Forbidden):
User "<user-name>" cannot create clusterrolebindings.rbac.authorization.k8s.io at the cluster scope.:
"Required \"container.clusterRoleBindings.create\" permission."
(post clusterrolebindings.rbac.authorization.k8s.io)
Any help would be much appreciated as this is blocking me from creating the permissions needed by my cluster services.
Janos's answer will work for GKE clusters that have been created with a password, but I'd recommend avoiding using that password wherever possible (or creating your GKE clusters without a password).
Using IAM: To create that ClusterRoleBinding, the caller must have the container.clusterRoleBindings.create permission. Only the OWNER and Kubernetes Engine Admin IAM Roles contain that permission (because it allows modification of access control on your GKE clusters).
So, to allow person#company.com to run that command, they must be granted one of those roles. E.g.:
gcloud projects add-iam-policy-binding $PROJECT \
--member=user:person#company.com \
--role=roles/container.admin
If your kubeconfig was created automatically by gcloud then your user is not the all powerful admin user - which you are trying to create a binding for.
Use gcloud container clusters describe <clustername> --zone <zone> on the cluster and look for the password field.
Thereafter execute kubectl --username=admin --password=FROMABOVE create clusterrolebinding ...

Google cloud dataproc failing to create new cluster with initialization scripts

I am using the below command to create data proc cluster:
gcloud dataproc clusters create informetis-dev
--initialization-actions “gs://dataproc-initialization-actions/jupyter/jupyter.sh,gs://dataproc-initialization-actions/cloud-sql-proxy/cloud-sql-proxy.sh,gs://dataproc-initialization-actions/hue/hue.sh,gs://dataproc-initialization-actions/ipython-notebook/ipython.sh,gs://dataproc-initialization-actions/tez/tez.sh,gs://dataproc-initialization-actions/oozie/oozie.sh,gs://dataproc-initialization-actions/zeppelin/zeppelin.sh,gs://dataproc-initialization-actions/user-environment/user-environment.sh,gs://dataproc-initialization-actions/list-consistency-cache/shared-list-consistency-cache.sh,gs://dataproc-initialization-actions/kafka/kafka.sh,gs://dataproc-initialization-actions/ganglia/ganglia.sh,gs://dataproc-initialization-actions/flink/flink.sh”
--image-version 1.1 --master-boot-disk-size 100GB --master-machine-type n1-standard-1 --metadata "hive-metastore-instance=g-test-1022:asia-east1:db_instance”
--num-preemptible-workers 2 --num-workers 2 --preemptible-worker-boot-disk-size 1TB --properties hive:hive.metastore.warehouse.dir=gs://informetis-dev/hive-warehouse
--worker-machine-type n1-standard-2 --zone asia-east1-b --bucket info-dev
But Dataproc failed to create cluster with following errors in failure file:
cat
+ mysql -u hive -phive-password -e '' ERROR 2003 (HY000): Can't connect to MySQL server on 'localhost' (111)
+ mysql -e 'CREATE USER '\''hive'\'' IDENTIFIED BY '\''hive-password'\'';' ERROR 2003 (HY000): Can't connect to MySQL
server on 'localhost' (111)
Does anyone have any idea behind this failure ?
It looks like you're missing the --scopes sql-admin flag as described in the initialization action's documentation, which will prevent the CloudSQL proxy from being able to authorize its tunnel into your CloudSQL instance.
Additionally, aside from just the scopes, you need to make sure the default Compute Engine service account has the right project-level permissions in whichever project holds your CloudSQL instance. Normally the default service account is a project editor in the GCE project, so that should be sufficient when combined with the sql-admin scopes to access a CloudSQL instance in the same project, but if you're accessing a CloudSQL instance in a separate project, you'll also have to add that service account as a project editor in the project which owns the CloudSQL instance.
You can find the email address of your default compute service account under the IAM page for your project deploying Dataproc clusters, with the name "Compute Engine default service account"; it should look something like <number>#project.gserviceaccount.com`.
I am assuming that you already created the Cloud SQL instance with something like this, correct?
gcloud sql instances create g-test-1022 \
--tier db-n1-standard-1 \
--activation-policy=ALWAYS
If so, then it looks like the error is in how the argument for the metadata is formatted. You have this:
--metadata "hive-metastore-instance=g-test-1022:asia-east1:db_instance”
Unfortuinately, the zone looks to be incomplete (asia-east1 instead of asia-east1-b).
Additionally, with running that many initializayion actions, you'll want to provide a pretty generous initialization action timeout so the cluster does not assume something has failed while your actions take awhile to install. You can do that by specifying:
--initialization-action-timeout 30m
That will allow the cluster to give the initialization actions 30 minutes to bootstrap.
By the time you reported, it was detected an issue with cloud sql proxy initialization action. It is most probably that such issue affected you.
Nowadays, it shouldn't be an issue.

gcloud compute ssh fails

I'm using Windows Server 2008.
on issuing gcloud compute ssh instance-1 --zone us-central1-a
I receive the error:
ERROR: (gcloud.compute.ssh) Could not fetch instance:
- Invalid value 'EFOnline'. Values must match the following regular expression: '(?:(?:[-a-z0-9]{1,63}\.)*(?:[a-z](?:[-
a-z0-9]{0,61}[a-z0-9])?):)?(?:[0-9]{1,19}|(?:[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?))'
To be clear, I did do a gcloud auth login and received a successful 'you are now authenticated'
My project Name is EFOnline
my instance name is : instance-1 us-central1-a (cut and paste there)
So why the weird regex error??
Thanks
Google Cloud projects have both a name and an id.
Your project ID is the string that uniquely identifies your project to Google. Project IDs show up in URI paths to cloud resources, and have to be "good" strings to put in URIs. You can find the project id for your project at the "Overview" in http://console.developers.google.com/ page.
The project name is a human-readable string that can, for example, contain spaces and some special characters.
GCloud (and most of the GCP tooling) uses project ID.
So... please try the following: Look up your project id here: http://console.developers.google.com/ then run
$ gcloud config set project <id>
$ gcloud compute ssh instance-1 --zone us-central1-a
Also, we're working on fixing the error message.
Also, if you have more google cloud accounts or you are not logged in, you need first to authenticate with google cloud:
gcloud auth login
after that, copy given link, login with wanted account and you will be able to SSH to google cloud console with gcloud command.
Additionally, one can log into their developer console, open up your VM instance and click on the SSH widget at the top of the page, selecting View Gcloud Command. This will generate your specific string for terminal.
gcloud compute --project "project_name" ssh --zone "us-central1-a" "vm_instance_name"