I have a very similar question to this one. #cherba already gave a very rich and helpful dissection of the gcloud init command which has been very helpful.
So what I really want to do, automating gcloud init is:
Front load my interactive input: I want the users to supply all input at the beginning and not be prompted again.
Request a token, before gcloud is even installed, probably from a static perma-link, the resulting token should be usable only once, probably with a limited lifetime, maybe an hour. This is very similar to how gcloud init —-console-only already works, except with an unchanging initial URL.
I specifically want this to be for a user account, not a service account.
This would allow me to prompt the user, upfront, for all configuration input, and build the fully configured system automatically, over lunch or a long coffee break; not needing additional babysitting.
The goal here is distinct development environments, not deploying to an array of boxes.
How can I accomplish this?
This is not supported officially and is not recommended. Service accounts are meant for this kind of thing. You should use service accounts as explained in the earlier answer.
What the SDK is essentially doing is submitting a token request to https://accounts.google.com/o/oauth2/auth with following scopes:
'https://www.googleapis.com/auth/userinfo.email'
'https://www.googleapis.com/auth/cloud-platform'
'https://www.googleapis.com/auth/appengine.admin'
'https://www.googleapis.com/auth/compute'
'https://www.googleapis.com/auth/accounts.reauth'
For this to succeed you need to provide the regular oauth parameters like client_id, client_secret. To generate these you will need to register your app as an oauth app in the developer console.
This may not work if third party authorizations are not supported. I have not tried it.
You said "Front load my interactive input:" and also "Request a token, before gcloud is even installed". The problem with your request above, is that you will need to install gcloud at some point in time and gcloud will use its own authentication methods to connect, meaning that authentication should happen after gcloud is installed because you will always use the command “gcloud ….” to somehow connect. The previous post that you linked explains this.
Due to this, I'm suspecting that you need a workflow where simultaneous gcloud commands will run on multiple users/projects at the same time, by running gcloud many times in parallel. As you know, Linux runs one command at a time and "front loading" the authentication (as you call it) can either be the "screen" command inside one SSH session or running multiple SSH sessions at the same time. If that's not what you need, then a simple shell script should do. The shell script will run commands one after the other rather than in parallel.
For example, let's say that you want to install a package that will take a long time and be able to run another command at the same time, then you could do the following:
$ screen
$ sudo apt-get install [package-name]
Press Ctrl-A” and “d“ to temporarily exit this session
$ … (do another process here)
$ screen -r (re-attaches screen to continue on previous process on line 2)
The example above is somewhat the equivalent of having multiple SSH sessions open at the same time. You could maybe open multiple “screens” and launch multiple authentications at the same time, thereby also controlling when you want to stop a session. Keep in mind that if you run things in parallel, you will definitely need to load the authentication file as mentioned in the post you linked. Otherwise, you can use simple shell scripting and pass arguments. Since i'm not sure of the process that comes before/after your authentication, it's hard for me to provide a more precise example. There's a lot to consider and many unknowns about your workflow. I've included references below that show all the possibilities.
References:
- https://www.linode.com/docs/networking/ssh/using-gnu-screen-to-manage-persistent-terminal-sessions/
- https://www.geeksforgeeks.org/screen-command-in-linux-with-examples/
- https://www.lifewire.com/pass-arguments-to-bash-script-2200571
- https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account
- https://cloud.google.com/sdk/gcloud/reference/auth/login
- https://cloud.google.com/sdk/docs/scripting-gcloud
Related
I know this has been asked many times because of the complete mess Google have made with authentication but I can't find an answer. I'm trying to create a CI pipeline that can use service account credentials from a file. I want to be able to run it locally or from a server. From what I've read gcloud inexplicably ignores the GOOGLE_APPLICATION_CREDENTIALS env var so I have to globally set my creds with the following, meaning I can kiss goodbye to any kind of parallelisation:
gcloud auth activate-service-account --key-file=$(GOOGLE_APPLICATION_CREDENTIALS)
Surely it must be possible to run multiple commands in parallel with different SA credentials?
Also, the above approach ignores the project ID specified in the key file, so gcloud tries to target the last project ID I personally set for myself.
Is there a solution to this ridiculousness? I'm looking for a non-interactive, non-destructive (i.e. won't trash my personal creds) way of calling gcloud in parallel with different service accounts and automatically using their project IDs. Is this possible?
Well it actually is possible with this:
CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE=$(GOOGLE_CREDENTIALS_FILE) \
CLOUDSDK_CORE_PROJECT=$(GCP_PROJECT) \
gcloud run deploy --allow-unauthenticated $(CLOUD_RUN_CONFIG) --image $(GCR_DOCKER_IMAGE)
It's a shame the docs are so poor it's taken me forever to find this info. Why gcloud doesn't just use the same env vars as all the libraries will remain a mystery to everyone outside Google...
I am learning puppet and am trying to write modules to install services such as tigervnc and openvpn.
The problem is that for tigervnc requires the initial password setting by the user. I have tried using:
"exec {'/usr/bin/echo password | /usr/bin/vncpasswd > ~/.vnc/passwd"
This works if I run it on the command line if I'm logged in as the user but does not work when run via puppet.
The problem with openvnc is that it requires a lot of user interaction for the default settings for certificate generation/certificate authority and key generation.
I have tried using execs with the "pkitool" methods which work to a point but not very well or stable. I am also wary of using many execs if there is a better way to do it.
So to sum up my main question is how to deal with these user interactions when trying to automate installations with puppet, and is there a better way than running lots of execs which to me seem like a last resort ?
Thanks
If setting up a piece of software requires user interaction, I don't really see a way around exec. Keeping its use to a minimum is indeed a sensible design goal.
An economic approach is to
create a script that does all the necessary lifting that Puppet resources cannot perform
make Puppet deploy that script to the agent
run it at appropriate times via exec (along with good creates or onlyif queries)
Scripts that run installation wizards that rely on interactive input should probably rely on expect and friends.
For a few days now the backup of google sites using google-sites-liberation stopped working.
The call
java -cp google-sites-liberation.jar com.google.sites.liberation.export.Main -d "$DOMAIN" -w wiki -u "$USER" -p "$PASSWORD" -f "$DIR/" 2>&1
which always worked before now fails with:
May 29, 2015 1:48:23 PM com.google.sites.liberation.export.Main doMain
SEVERE: Invalid User Credentials!
Exception in thread "main" java.lang.RuntimeException: com.google.gdata.util.AuthenticationException: Error authenticating (check service name)
at com.google.sites.liberation.export.Main.doMain(Main.java:89)
at com.google.sites.liberation.export.Main.main(Main.java:97)
Caused by: com.google.gdata.util.AuthenticationException: Error authenticating (check service name)
at com.google.gdata.client.GoogleAuthTokenFactory.getAuthException(GoogleAuthTokenFactory.java:614)
at com.google.gdata.client.GoogleAuthTokenFactory.getAuthToken(GoogleAuthTokenFactory.java:490)
at com.google.gdata.client.GoogleAuthTokenFactory.setUserCredentials(GoogleAuthTokenFactory.java:336)
at com.google.gdata.client.GoogleService.setUserCredentials(GoogleService.java:362)
at com.google.gdata.client.GoogleService.setUserCredentials(GoogleService.java:317)
at com.google.gdata.client.GoogleService.setUserCredentials(GoogleService.java:301)
at com.google.sites.liberation.export.Main.doMain(Main.java:79)
... 1 more
I checked the credentials, the credentials of the account are correct. However it is the main account's password, which probably has more strict security settings on Google now.
I tried to find a solution using Google-Search but only stumbled over old suggestions which had solutions which are no more available today. Also I did not find a way to add an user/password application login to the account used to backup the wiki.
Has anybody a pointer how to fix that and make backup of google site available again?
All answers are good which offer a solution to backup a site:
Use some other fully^2 automated tool which does the job of copying an entire site to a directory or archive format, for example .tar.bz2
Change google-sites-liberation such, that it uses another authentication method then given in the docs which are a couple of years old now. I did not manage to find it.
Note that the account used for backup must not have full google apps for domains administrator access, as this is crucial.
Please no external vendor links except if it is from Google. The data of the site(s) must not be shared with a third party, only Google and me.
Note that the process must be fully^2 automated, but I would like to have it even fully^4 automated:
fully^1, because it must run at regular intervals.
fully^2, because it must start without user intervention whatsoever (some people define "fully automated" as to start something manually such that it runs by itself, while "automated" means to have a script which still may ask for some additional input)
fully^3, because it should not involve user intervention to get the process started (like issuing something like a google authenticator token) at the first run (even if it later runs fully^2 automated)
fully^4, because I want to be able to setup the process for several thousands sites in an automated, noninteractive way, when the process which prepares the setup runs on a host which is offline (so the setup can be uploaded to the fully^3 automated system without any additional manual setup steps for example using IPoAC. YKWIM).
Not much of a problem if it is only fully^2 automated, as I only want to backup my little single site (only a few thousand pages with attachments). However I am curious how to get it fully^4 automated, because automating everything (including, but not limited to, the Universe) was my motivation getting into the computer business several decades ago ..
Thanks.
Links:
https://code.google.com/p/google-sites-liberation/ a bit dated code to retrieve sites
https://www.google.com/settings/takeout does not include google apps for domain sites
http://blog.famzah.net/2014/08/06/authentication-for-google-sites-liberation/ the noted account setting is not (no more) available
Was unable to find any suitable link how to implement a google apps for domain backup with another tool, the all result pages I looked at (several!) seem to be exclusively for third party vendors on this matter with more or less unknown trustworthyness. So perhaps I am unable to define the right google search on this matter.
Update 2015-06-23:
My scripts run every day and they tell if something goes wrong, but not if they work as intended. So I oversaw that it suddenly worked for a few days. But today it failed again:
2015-05-27 to 2015-06-11 (15 days) authentication failure
2015-06-12 to 2015-06-22 (11 days) it works again
2015-06-23 (today) authentication failure again
I have no idea why it suddenly worked for 11 days. I'll probably update this question again on the next ok-to-fail transition. ;)
Google uses OAuth2 instead of user account/password.
I fixed the GUI interface.
https://github.com/sih4sing5hong5/google-sites-liberation
But I have no idea about OAuth2 with auto scripts.
I developed a console script in Python which exports Google Sites:
https://github.com/famzah/google-sites-backup
This works with automated scripts. It needs more testing but functions properly for my sites.
Because of the nature of OAuth2, the first time you ever start the script, you will need to obtain a token manually by visiting a web page. There is no other way. Once you've done this, the Python script caches the authentication token and the backup works in a completely non-interactive mode. It is a decision by Google when this cached token expires.
Hey guys so I have an xcode bot problem. Basically I have a bot that requires a pre-script to be ran. This script runs the command a git submodule init and update, and gets an ssh authentication error.
On the os x server machine it self the appropriate ssh keys have been set on the user admin (tested). In xcode on my machine the server is connect as the user admin. However it seems like when the script is being run it is not being run as admin(tested by creating a text file in ~ and wasnt there after). I was wondering if it was possible to su in the script, i've looked online and it seems like it wouldn't be possible because I don't know what user xcode bot is running the script as (my guess is its running as Guess)
Any advice on this? Or on a way to run the command as a different user(must be in the script)
_xcsbuildd is the account which the bot runs under. Make sure that account has the necessary permissions.
Xcode server runs an integration from a separate user called _xcsbuildd. If it is possible for you to login remotely to the machine that Xcode Server is running then you can login under that user through the Terminal, and should be able to add or check any ssh keys that are loaded with that user.
Here is a useful blog post on how to do this http://papaanton.com/setting-up-xcode-6-and-apple-server-4-0-for-continues-integration-with-cocoapods/
Scroll down to the part called Adding additional SSH Key to the Xcode Server That should be able to walk you through how to do that. I know its not an automated script, but its how I was able to get past my SSL issues, and maybe it'll help you as well
I need to allow a PHP script on my local web server, to SSH to another machine to perform a specified task on some files. My httpd runs as _www with low permissions, so setting up direct passwordless SSH is difficult, not to say ill-advised.
The way I do it now is to have a minimal PHP script that sudo-exec's (as me) a shell script which is outside of the document root. The shell script in turn calls (as me) the PHP code that does the actual SSH work, and prints its output. Here's the code.
read_remote_files.php (The script I call from my browser):
exec('sudo -u me -n /home/me/run_php.sh /path/to/my_prog.php', $results);
print $results;
/home/me/run_php.sh (Runs as me, calls whatever it's given):
php $1 2>&1
sudoers:
_www ALL = (me) NOPASSWD: /home/me/run_php.sh
This all works, as my_prog.php is called as me and can SSH as me. It seems it's not too insecure since run_php.sh can't be called directly from a browser (outside document root). The issue I'm having is that my_prog.php isn't called as an HTTP program so doesn't have access to the HTTP environment variables (DOCUMENT_ROOT etc).
Two questions:
Am I making this too complicated?
Is there an easy way for my final script to get the HTTP variables?
Thanks!
Andy
Many systems do stuff like this using a (privileged) cron job that frequently checks for the existence of a file, a database record or some other resource, and then performs actions if there are any.
The huge advantage of this is that there is no direct interaction between the PHP script and the privileged script at all. The PHP script leaves the instructions in a resource, the privileged script fetches it. As long as the instructions can't lead to the system getting compromised or damaged, it's definitely more secure than sudoing.
The disadvantage is that you can't push changes whenever you like; you have to wait until the cron job runs again. But maybe it's an option anyway?
"I need to allow a PHP script on my local web server, to SSH to another machine to perform a specified task on some files."
I think that you are phrasing this in terms of a solution that you have difficulty in getting to work rather than a requirement. Surely what you should be saying is "I want to invoke a task on machine B from a PHP script running under Apache on Machine A." And then research solutions to this -- to which there are many from a simple 'roll-your-own' RPC tunnelled over HTTP(S) to using an XMLRPC or SOA framework.
Two caveats:
Do a phpinfo(); on both machines to check what extensions are available and
Also check your php.ini setting to make sure that your service provider hasn't disabled any functions that you expect to use (or do a Q&D script to echo 'disable_functions = ' . ini_get('disable_functions') . "\n"; ...)
If you browse here and the wider internet you'll find many examples. Here is one that I use for a similar purpose.