I have installed the TeamCity server on my infrastructure. I was e.g. able to silently configure the external database and some other stuff through the TeamCity configuration files. Now, when I start TeamCity, I am facing the "TeamCity First Start" web page, where I can choose if I want to restore my system from a backup or if I want to proceed with a First Start.
I would like to automate that installation as much as possible. Is there a reliable way, e.g. through the TeamCity API, to trigger the action underlying the "Proceed" button of that web page, other than writing a bot that browses the web page and clicks the button? I found no such thing in the API documentation, but I might have missed something.
I think something like this could be called on the server host:
curl -X POST http://localhost:8111/mnt/do/goNewInstallation
but I don't know what data to post and how to get the necessary authentication to be allowed to run that command. Indeed, running the above command yields the following error:
The session is not authenticated. Access denied.
At that time, the super user authentication token has not yet been displayed to the server log file.
Related
I have a very similar question to this one. #cherba already gave a very rich and helpful dissection of the gcloud init command which has been very helpful.
So what I really want to do, automating gcloud init is:
Front load my interactive input: I want the users to supply all input at the beginning and not be prompted again.
Request a token, before gcloud is even installed, probably from a static perma-link, the resulting token should be usable only once, probably with a limited lifetime, maybe an hour. This is very similar to how gcloud init —-console-only already works, except with an unchanging initial URL.
I specifically want this to be for a user account, not a service account.
This would allow me to prompt the user, upfront, for all configuration input, and build the fully configured system automatically, over lunch or a long coffee break; not needing additional babysitting.
The goal here is distinct development environments, not deploying to an array of boxes.
How can I accomplish this?
This is not supported officially and is not recommended. Service accounts are meant for this kind of thing. You should use service accounts as explained in the earlier answer.
What the SDK is essentially doing is submitting a token request to https://accounts.google.com/o/oauth2/auth with following scopes:
'https://www.googleapis.com/auth/userinfo.email'
'https://www.googleapis.com/auth/cloud-platform'
'https://www.googleapis.com/auth/appengine.admin'
'https://www.googleapis.com/auth/compute'
'https://www.googleapis.com/auth/accounts.reauth'
For this to succeed you need to provide the regular oauth parameters like client_id, client_secret. To generate these you will need to register your app as an oauth app in the developer console.
This may not work if third party authorizations are not supported. I have not tried it.
You said "Front load my interactive input:" and also "Request a token, before gcloud is even installed". The problem with your request above, is that you will need to install gcloud at some point in time and gcloud will use its own authentication methods to connect, meaning that authentication should happen after gcloud is installed because you will always use the command “gcloud ….” to somehow connect. The previous post that you linked explains this.
Due to this, I'm suspecting that you need a workflow where simultaneous gcloud commands will run on multiple users/projects at the same time, by running gcloud many times in parallel. As you know, Linux runs one command at a time and "front loading" the authentication (as you call it) can either be the "screen" command inside one SSH session or running multiple SSH sessions at the same time. If that's not what you need, then a simple shell script should do. The shell script will run commands one after the other rather than in parallel.
For example, let's say that you want to install a package that will take a long time and be able to run another command at the same time, then you could do the following:
$ screen
$ sudo apt-get install [package-name]
Press Ctrl-A” and “d“ to temporarily exit this session
$ … (do another process here)
$ screen -r (re-attaches screen to continue on previous process on line 2)
The example above is somewhat the equivalent of having multiple SSH sessions open at the same time. You could maybe open multiple “screens” and launch multiple authentications at the same time, thereby also controlling when you want to stop a session. Keep in mind that if you run things in parallel, you will definitely need to load the authentication file as mentioned in the post you linked. Otherwise, you can use simple shell scripting and pass arguments. Since i'm not sure of the process that comes before/after your authentication, it's hard for me to provide a more precise example. There's a lot to consider and many unknowns about your workflow. I've included references below that show all the possibilities.
References:
- https://www.linode.com/docs/networking/ssh/using-gnu-screen-to-manage-persistent-terminal-sessions/
- https://www.geeksforgeeks.org/screen-command-in-linux-with-examples/
- https://www.lifewire.com/pass-arguments-to-bash-script-2200571
- https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account
- https://cloud.google.com/sdk/gcloud/reference/auth/login
- https://cloud.google.com/sdk/docs/scripting-gcloud
I have been trying to get duplicity to backup to google drive. But it looks like it is still using the old client API.
I found some thread saying that the new API should be supported but not much details on how to get it to work.
I got as far as compiling and using duplicity 7.0.3 but then I got this error:
BackendException: GOOGLE_DRIVE_ACCOUNT_KEY environment variable not set. Please read the manpage to fix.
Has anyone set up duplicity to work with Google Drive and know how to do this?
Now that Google has begun forcing clients to use OAuth, using Google Drive as a backup target has actually gotten very confusing. I found an excellent blog post that walked me through it. The salient steps are:
Install PyDrive
PyDrive is the library that lets Duplicity use OAuth to access Drive.
pip install pydrive
should be sufficient, or you can go through your distribution's package manager.
Create an API token
Navigate to the Google Developer Console and log in. Create a project and select it from the drop-down on the top toolbar.
Now select the "Enable APIs and Services" button in the Dashboard, which should already be pulled up, but if not, is in the hamburger menu on the left.
Search for and enable the Drive API. After it's enabled, you can actually create the token. Choose "Credentials" from the left navigation bar, and click "Add Credential" > "OAuth 2.0 Client ID." Set the application type to "Other."
After the credential is created, click on it to view the details. Your Client ID and secret will be displayed. Take note of them.
Configure Duplicity
Whew. Time to actually configure the program. Paste the following into a file, replacing your client ID and secret with the ones from the Console above.
client_config_backend: settings
client_config:
client_id: <your client ID>.apps.googleusercontent.com
client_secret: <your client secret>
save_credentials: True
save_credentials_backend: file
save_credentials_file: gdrive.cache
get_refresh_token: True
(I'm using the excellent Duply frontend, so I saved this as ~/.duply/<server name>/gdrive).
Duplicity needs to be given the name of this file in the GOOGLE_DRIVE_SETTINGS environment variable. So you could invoke duplicity like this:
GOOGLE_DRIVE_SETTINGS=gdrive duplicity <...>
Or if you're using Duply, you can export this variable in the Duply configuration file:
export GOOGLE_DRIVE_SETTINGS=gdrive
Running Duplicity for the first time will begin the OAuth process; you'll be given a link to visit, which will ask permission for the app you created earlier in the Console to access your Drive account. Accept, and it will give you another authentication token to paste back into the terminal. The authorization info will be saved in a .cache file alongside the gdrive settings file.
At this point you should be good to go, and Duplicity should behave normally. Good luck!
For a few days now the backup of google sites using google-sites-liberation stopped working.
The call
java -cp google-sites-liberation.jar com.google.sites.liberation.export.Main -d "$DOMAIN" -w wiki -u "$USER" -p "$PASSWORD" -f "$DIR/" 2>&1
which always worked before now fails with:
May 29, 2015 1:48:23 PM com.google.sites.liberation.export.Main doMain
SEVERE: Invalid User Credentials!
Exception in thread "main" java.lang.RuntimeException: com.google.gdata.util.AuthenticationException: Error authenticating (check service name)
at com.google.sites.liberation.export.Main.doMain(Main.java:89)
at com.google.sites.liberation.export.Main.main(Main.java:97)
Caused by: com.google.gdata.util.AuthenticationException: Error authenticating (check service name)
at com.google.gdata.client.GoogleAuthTokenFactory.getAuthException(GoogleAuthTokenFactory.java:614)
at com.google.gdata.client.GoogleAuthTokenFactory.getAuthToken(GoogleAuthTokenFactory.java:490)
at com.google.gdata.client.GoogleAuthTokenFactory.setUserCredentials(GoogleAuthTokenFactory.java:336)
at com.google.gdata.client.GoogleService.setUserCredentials(GoogleService.java:362)
at com.google.gdata.client.GoogleService.setUserCredentials(GoogleService.java:317)
at com.google.gdata.client.GoogleService.setUserCredentials(GoogleService.java:301)
at com.google.sites.liberation.export.Main.doMain(Main.java:79)
... 1 more
I checked the credentials, the credentials of the account are correct. However it is the main account's password, which probably has more strict security settings on Google now.
I tried to find a solution using Google-Search but only stumbled over old suggestions which had solutions which are no more available today. Also I did not find a way to add an user/password application login to the account used to backup the wiki.
Has anybody a pointer how to fix that and make backup of google site available again?
All answers are good which offer a solution to backup a site:
Use some other fully^2 automated tool which does the job of copying an entire site to a directory or archive format, for example .tar.bz2
Change google-sites-liberation such, that it uses another authentication method then given in the docs which are a couple of years old now. I did not manage to find it.
Note that the account used for backup must not have full google apps for domains administrator access, as this is crucial.
Please no external vendor links except if it is from Google. The data of the site(s) must not be shared with a third party, only Google and me.
Note that the process must be fully^2 automated, but I would like to have it even fully^4 automated:
fully^1, because it must run at regular intervals.
fully^2, because it must start without user intervention whatsoever (some people define "fully automated" as to start something manually such that it runs by itself, while "automated" means to have a script which still may ask for some additional input)
fully^3, because it should not involve user intervention to get the process started (like issuing something like a google authenticator token) at the first run (even if it later runs fully^2 automated)
fully^4, because I want to be able to setup the process for several thousands sites in an automated, noninteractive way, when the process which prepares the setup runs on a host which is offline (so the setup can be uploaded to the fully^3 automated system without any additional manual setup steps for example using IPoAC. YKWIM).
Not much of a problem if it is only fully^2 automated, as I only want to backup my little single site (only a few thousand pages with attachments). However I am curious how to get it fully^4 automated, because automating everything (including, but not limited to, the Universe) was my motivation getting into the computer business several decades ago ..
Thanks.
Links:
https://code.google.com/p/google-sites-liberation/ a bit dated code to retrieve sites
https://www.google.com/settings/takeout does not include google apps for domain sites
http://blog.famzah.net/2014/08/06/authentication-for-google-sites-liberation/ the noted account setting is not (no more) available
Was unable to find any suitable link how to implement a google apps for domain backup with another tool, the all result pages I looked at (several!) seem to be exclusively for third party vendors on this matter with more or less unknown trustworthyness. So perhaps I am unable to define the right google search on this matter.
Update 2015-06-23:
My scripts run every day and they tell if something goes wrong, but not if they work as intended. So I oversaw that it suddenly worked for a few days. But today it failed again:
2015-05-27 to 2015-06-11 (15 days) authentication failure
2015-06-12 to 2015-06-22 (11 days) it works again
2015-06-23 (today) authentication failure again
I have no idea why it suddenly worked for 11 days. I'll probably update this question again on the next ok-to-fail transition. ;)
Google uses OAuth2 instead of user account/password.
I fixed the GUI interface.
https://github.com/sih4sing5hong5/google-sites-liberation
But I have no idea about OAuth2 with auto scripts.
I developed a console script in Python which exports Google Sites:
https://github.com/famzah/google-sites-backup
This works with automated scripts. It needs more testing but functions properly for my sites.
Because of the nature of OAuth2, the first time you ever start the script, you will need to obtain a token manually by visiting a web page. There is no other way. Once you've done this, the Python script caches the authentication token and the backup works in a completely non-interactive mode. It is a decision by Google when this cached token expires.
Hey guys so I have an xcode bot problem. Basically I have a bot that requires a pre-script to be ran. This script runs the command a git submodule init and update, and gets an ssh authentication error.
On the os x server machine it self the appropriate ssh keys have been set on the user admin (tested). In xcode on my machine the server is connect as the user admin. However it seems like when the script is being run it is not being run as admin(tested by creating a text file in ~ and wasnt there after). I was wondering if it was possible to su in the script, i've looked online and it seems like it wouldn't be possible because I don't know what user xcode bot is running the script as (my guess is its running as Guess)
Any advice on this? Or on a way to run the command as a different user(must be in the script)
_xcsbuildd is the account which the bot runs under. Make sure that account has the necessary permissions.
Xcode server runs an integration from a separate user called _xcsbuildd. If it is possible for you to login remotely to the machine that Xcode Server is running then you can login under that user through the Terminal, and should be able to add or check any ssh keys that are loaded with that user.
Here is a useful blog post on how to do this http://papaanton.com/setting-up-xcode-6-and-apple-server-4-0-for-continues-integration-with-cocoapods/
Scroll down to the part called Adding additional SSH Key to the Xcode Server That should be able to walk you through how to do that. I know its not an automated script, but its how I was able to get past my SSL issues, and maybe it'll help you as well
I am kind of confused. So pls go easy on me. Take any standard web application implemented with mvc, like codeigniter or rails. The scripts gets executed only when a browser sends request right. So when a user logs in and sends request the server recieves it and sends him response.
Now consider the scenario where apart from the regular application i also need something like a backend process. For example a script which checks whether a bidding time is closed and sends the mail to the bidder that the bidding is closed and chooses the bid winner. Now all these actions has to be done automatically as soon as the bidding time ends.
Now if this script is part of a regular app then it should be triggered by the client(browser) but i dont want that to happen. This should be like a bot script which must run on the server checking the DB for events and patterns like this.
How do i go about doing something like this. Also is this possible to have this implemented on a regular shared or dedicated hosting where we dont have shell access but only ftp access.
You'd have to write your script as a standalone program and either have it run continuously in the background or have cron (or some other scheduling service; also only works if you're only interested in time-based events) execute it for you.
There are probably hosts that have shell-less ways to do this (fancy GUI interfaces for managing background processes or something,) but your run of the mill web host with only FTP access definitely doesn't.
You need a cron job, it's easy to set up on linux. That cron job will either call the command line version of PHP with your script or create a local HTTP request with curl or wget.
If you don't have access then you need an external site that automatically generates periodic HTTP requests. A cheap one is setcronjob.