This question already has answers here:
How do I set an env var with a bash expression in GitHub Actions?
(2 answers)
Closed 3 years ago.
With Github Actions how to do a backup of a file?
I need to create a new folder named with the date/time of today.
name: Backup Database
on:
push:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Backup
uses: appleboy/ssh-action#master
with:
host: ${{ secrets.HOST }}
username: ${{ secrets.USERNAME }}
key: ${{ secrets.KEY }}
port: 22
script: |
DATE=${{ NEW DATE }} # Need to create a variable with date/time
mkdir backup_${{ DATE }} # Create a new folder with the date/time
This user found a hack to create vars that are generated with bash commands output.
In his words:
I found a way to hack this limitation. Write your VAR on disk (the CI
system disk), then cat $my_var to use your VAR in every step you need
Reference: "How to define env variable?"
Related
Something changed in the way the dotnet publish workflow task works. We've been using this pretty straightforward yaml script for some time now.
name: Publish to staging server
env:
AZURE_WEBAPP_NAME: 'my-dotnet-webapp'
AZURE_SLOT_NAME: 'staging'
GITHUB_PUBLISH_SECRET: ${{ secrets.AZURE_DEPLOYMENTSLOT_STAGING }}
AZURE_WEBAPP_PACKAGE_PATH: '.'
DOTNET_VERSION: '7.0.0'
on:
push:
branches:
- staging
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Set up .NET Core
uses: actions/setup-dotnet#v3
with:
dotnet-version: ${{ env.DOTNET_VERSION }}
- name: Set up dependency caching for faster builds
uses: actions/cache#v3
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/packages.lock.json') }}
restore-keys: |
${{ runner.os }}-nuget-
- name: Build with dotnet
run: dotnet build --configuration Release
- name: dotnet publish
run: dotnet publish -c Release -o ${{env.DOTNET_ROOT}}/myapp
- name: Upload artifact for deployment job
uses: actions/upload-artifact#v3
with:
name: .net-app
path: ${{env.DOTNET_ROOT}}/myapp
retention-days: 1
deploy:
runs-on: ubuntu-latest
needs: build
environment:
name: 'Production'
url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
steps:
- name: Download artifact from build job
uses: actions/download-artifact#v3
with:
name: .net-app
- name: Deploy to Azure Web App
id: deploy-to-webapp
uses: azure/webapps-deploy#v2
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
slot-name: ${{ env.AZURE_SLOT_NAME }}
publish-profile: ${{ env.GITHUB_PUBLISH_SECRET }}
package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}
Today, I tried to run with workflow and received the following error during the dotnet publish step:
Error: /usr/share/dotnet/sdk/7.0.200/Current/SolutionFile/ImportAfter/Microsoft.NET.Sdk.Solution.targets(36,5): error NETSDK1194: The "--output" option isn't supported when building a solution.
I expected the workflow to run without error as it has done dozens of times previously.
What's really going on here?
After a considerable amount of research and a little trial and error, I realized that I had to explicitly specify the webapp project file as an argument of the command. This is because I do still need to use the output option, so Github knows where to find the files in the subsequent deploy workflow.
Then there was the matter of figuring out the file path for the project file. This may vary for others based on their specific Visual Studio solution file structure.
Here is the fix that worked to resolve this issue (assume that when I created my project in VS, I named it MyWebApp:
- name: dotnet publish
run: dotnet publish ~/work/MyWebApp/MyWebApp/MyWebApp/MyWebApp.csproj -c Release -o ${{env.DOTNET_ROOT}}/myapp
Yes, that's 3 directories deep. The project file in my Windows file explorer is only 2 directories deep.
Hope this helps someone.
The reason for this is a update by .NET according to This guy, what worked for me was changing
--output .
To
--property:PackageOutputPath=.
I want to update ONLY the first header of a readme so it always has the repo's name.
I know how to get the repo name in github actions by doing something like:
name: CI
on:
push:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Run a one-line script
run: echo "REPO_NAME=${{ github.event.repository.name }}" >> $GITHUB_ENV
However, I want to access my readme.md and add the 'github.event.repository.name' to the header.
So I would make a Readme with something like:
Introduction for: {{ github.event.repository.name }}
hoping I can get something like this with gitactions:
Introduction for: RepoName
I tried using some marketplace github actions but the one I tried seems to not do variables and it seems to update the whole readme.md not just the header: https://github.com/marketplace/actions/dynamic-readme
Here is the failed example for this marketplace plugin:
name: Dynamic Template
on:
push:
branches:
- main
workflow_dispatch:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Fetching Repository Contents"
uses: actions/checkout#main
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo "REPO_NAME=${{ github.event.repository.name }}" >> $GITHUB_ENV
# Runs a set of commands using the runners shell
- name: Update GitHub Profile README
uses: theboi/github-update-readme#v1.0
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO_NAME: ${{ github.event.repository.name }}
with:
header: $REPO_NAME
Is there any way to make the readme.md file have the repo name dynamically with a variable in github actions?
EDIT: I think I am very close but I don't know how to commit code in github actions. I figured, I can do this manually by using the sed command in bash in github actions. I think it worked but I think I have to commit the code to make it save. Here is the code I have so far:
name: Dynamic Template
on:
push:
branches:
- main
workflow_dispatch:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Fetching Repository Contents"
uses: actions/checkout#main
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo "REPO_NAME=${{ github.event.repository.name }}" >> $GITHUB_ENV
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
ls
sed -i 's/<reponame>/$REPO_NAME/' README.md
cat README.md
echo $REPO_NAME
I figured it out. You can it manually by using the sed command in bash within a runner in github actions. Set your README.md with a variable that you want to replace like <reponame> then use github actions to find that string and replace it with something you want (for me the repo name).
name: Dynamic Template
on:
create:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Fetching Repository Contents"
uses: actions/checkout#main
# Runs a set of commands using the runners shell
- name: Update README.md
run: |
sed -i 's/<reponame>/'${{ github.event.repository.name }}'/' README.md
git config user.email "41898282+github-actions[bot]#users.noreply.github.com"
git config user.name "github-actions[bot]"
git commit -am "Automated report"
git push
The email I used is a dependabot mentioned from here: https://github.com/orgs/community/discussions/26560#discussioncomment-3531273
This method needs four files
README.md
python script #replace tag
github_action.yml #run it automatically
variable.json #define variable
#PythonKiddieScripterX provides a good idea.
Based on the idea that context inside <> those <information will be hidden>. We can update our readme files by replacing those tags <> with the texts we want.
Example of my README.md file
This answer is version `1.0`. It will be updated automatically by loading `json` file and using GitHub Action.
In my readme.json file
{
VERSION: 1.0
}
Then I will only need a script to do the replacement. In this example, I use python saved as
replace_tag.py.
import re
import os
import json
# read all .md files
readmefiles = []
for root, dirs, files in os.walk("."):
for file in files:
if file.endswith(".md"):
readmefiles.append(os.path.join(root, file))
# load variable json
with open('readme.json') as f:
var_dic = json.load(f)
# match pattern (<variable-*.?-tag>)(`*.?`)
# example: (<variable-VERSION-tag>)(`1.1`)
for filename in readmefiles:
with open(filename,"r") as f:
content = f.read()
# update readme variables
for key, value in var_dic.items():
pattern = r"(<variable-{}-tag>)(`.*?`)".format(key)
replacement = r"\1`{}`".format(value)
content = re.sub(pattern, replacement, content)
with open(filename,"w") as f:
f.write(content)
Then the final thing is to use GitHub Action to run this python script every time there is a change.
yml file could be
name: README Dynamic Update
on:
push:
paths:
- *.md
- readme.json
- **/*.md
workflow_dispatch:
jobs:
update_templates:
name: "Update Templates"
runs-on: ubuntu-latest
steps:
- name: "📥 Update GitHub Readme files"
uses: actions/checkout#main
# Runs a set of commands using the runners shell
- name: Update README.md
run: |
python replace_tag.py
- name: pull-request
uses: repo-sync/pull-request#v2
with:
destination_branch: "main"
github_token: ${{ secrets.GITHUB_TOKEN }}
- name: commit
run: |
git config --global user.email youremail
git config --global user.name yourusername
git add .
git commit -m "README update Automation" -a
- name: Push changes
uses: ad-m/github-push-action#master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
Also for those text inside the code blocks, we can use <pre></pre> to solve the hidding issue.
Note that
need to add tag in this format <variable-YOURTAG-tag>"variable"
This question came from fact that you can only use global variables for deployment:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops#schema
Use only global level variables for defining a pool name. Stage/job level variables are not supported to define pool name.
But my config is a bit more complex. I do have main pipeline.yml with several stages. Stage's jobs defined in the template, so I have following structure:
# pipeline.yml (some cuts)
....
stages:
- stage: Build
displayName: Build all
jobs:
- template: ../templates/build.yml
parameters:
stage: 'DEV'
- stage: Dev
displayName: Deploy Dev
variables:
- group: GROUP-DEV
pool:
name: '$(env._global.agentPool)' # env._global.agentPool defined in in the variable group
jobs:
- template: ../templates/deployment.yml
parameters:
stage: 'DEV'
# other stages here ....
# templates/deployment.yml
parameters:
- name: stage
displayName: 'Stage'
type: string
jobs:
- deployment: Deploy${{ parameters.stage }}
displayName: Deploy ${{ parameters.stage }}
environment: ${{ parameters.stage }}
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: art_${{ parameters.stage }}
displayName: 'Download art_${{ parameters.stage }}'
# more steps here.....
Now the problem: with such config I got error:
##[error]Pipeline does not have permissions to use the referenced pool(s) . For authorization details, refer to https://aka.ms/yamlauthz.
which means we faced with limitation mentioned in the first quote (proof: https://developercommunity.visualstudio.com/t/pool-name-in-deployment-job-cannot-be-in-a-variabl/940174)
As soon as I change either pool name to hardcoded value or remove -deployment and use regular steps all works fine.
Question: with that, is there any chance to re-work templates to still use agent pool name from group variable? I need the -deployment task to reference environment.
What was tried already:
use pool name as param of the templates/deployment.yml and use pool inside steps. No luck, same story
I'm working on 2 github actions workflows:
Train a model and save it to s3 (monthly)
Download the model from s3 and use it in predictions (daily)
Using https://github.com/jakejarvis/s3-sync-action I was able to complete the first workflow. I train a model and then sync a dir, 'models' with a bucket on s3.
I had planned on using the same action to download the model for use in prediction but it looks like this action is one directional, upload only no download.
I found out the hard way by creating a workflow and attempting to sync with the runner:
retreive-model-s3:
runs-on: ubuntu-latest
steps:
- name: checkout current repo
uses: actions/checkout#master
- name: make dir to sync with s3
run: mkdir models
- name: checkout s3 sync action
uses: jakejarvis/s3-sync-action#master
with:
args: --follow-symlinks
env:
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_S3_ENDPOINT: ${{ secrets.AWS_S3_ENDPOINT }}
AWS_REGION: 'us-south' # optional: defaults to us-east-1
SOURCE_DIR: 'models' # optional: defaults to entire repository
- name: dir after
run: |
ls -l
ls -l models
- name: Upload model as artifact
uses: actions/upload-artifact#v2
with:
name: xgb-model
path: models/regression_model_full.rds
At the time of running, when I login to the UI I can see the object regression_model_full.rds is indeed there, it's just not downloading. I'm still unsure if this is expected or not (the name of the action 'sync' is what's confusing me).
For our s3 we must use the parameter AWS_S3_ENDPOINT. I found another action, AWS S3 here but unlike the sync action I started out with there's no option to add AWS_S3_ENDPOINT. Looking at the repo too it's two years old except a update tot he readme 8 months ago.
What's the 'prescribed' or conventional way to download from s3 during a workflow?
Soo I had the same problem as you. I was trying to download from S3 to update a directory folder in GitHub.
What I learned from actions is if you're updating some files in the repo you must follow the normal approach as if you were doing it locally eg) checkout, make changes, push.
So for your particular workflow you must checkout your repo in the workflow using actions/checkout#master and after you sync with a particular directory the main problem I was not doing was then pushing the changes back to the repo! This allowed me to update my folder daily.
Anyway, here is my script and hope you find it useful. I am using the AWS S3 action you mention towards the end.
# This is a basic workflow to help you get started with Actions
name: Fetch data.
# Controls when the workflow will run
on:
schedule:
# Runs "at hour 6 past every day" (see https://crontab.guru)
- cron: '00 6 * * *'
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: keithweaver/aws-s3-github-action#v1.0.0 # Verifies the recursive flag
name: sync folder
with:
command: sync
source: ${{ secrets.S3_BUCKET }}
destination: ./data/
aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws_region: ${{ secrets.AWS_REGION }}
flags: --delete
- name: Commit changes
run: |
git config --local user.email "action#github.com"
git config --local user.name "GitHub Action"
git add .
git diff-index --quiet HEAD || git commit -m "{commit message}" -a
git push origin main:main
Sidenote: the flag --delete allows you to keep your current folder up to date with your s3 folder by deleting any files that are not present in your s3 folder anymore
I'm in the throes of setting up a Github Action that should run an SSH command to connect to a private server. The private server's connection settings i have specify an identityFile, which I do own. After this connection, I will then run a proxycommand, so this is essentially to a bastion, for context.
What I cannot quite figure out at this point is how/which github action supports this configuration. I see the commands on this one (similar to others): https://github.com/appleboy/ssh-action/blob/master/action.yml and no mention of identifyFile property. Is there another way to execute this or a ssh command that can make this possible?
Would appreciate some pointers, thanks!
If you need some explanation of how to write your action, you can read this article : How to create Github Actions to run tests with docker services .
You just have to create your workflow file and use the actions of appleboy like on steps keyword :
- name: executing remote ssh commands using password
uses: appleboy/ssh-action#master
with:
host: ${{ secrets.HOST }}
username: ${{ secrets.USERNAME }}
key: ${{ secrets.KEY }}
key_path: ${{ secrets.KEY_PATH }}
password: ${{ secrets.PASSWORD }}
port: ${{ secrets.PORT }}
script: whoami
With the script line, you can execute what you want to do in the server and connect with the parameters set above. For multiple line do like this :
script: |
pwd
ls -al
Hope it will help.