I have a couple of integration tests which use connection strings. The test is run from an azure devops pipeline, with yaml definition (DotNetCoreCLI#2 task)
This connstrings used to be set in the .runsettings file, but I would rather they were secret, so I am moving the conn string to the pipeline library, and marking as secret.
Meanwhile, I have removed the secrets from the runsettings file, leaving the parameter values as empty strings there.
So now, I just have to override the test run parameters value when running dotnet test...
From reading the docs from Microsoft here, I have verified that I can accomplish this in Powershell
dotnet test path\to\mytest.dll --logger trx --settings path\to\my.runsettings --% -- TestRunParameters.Parameter(name=\"Customer1.TestConnStr\", value=\"Id=xxx;Pass=yyy\") TestRunParameters.Parameter(name=\"Customer2.TestConnStr\", value=\"Id=xxx;Pass=yyy\")
I also see that the --% is not used when running from cmd, which I think is more appropriate for my yaml pipeline (running on a windows agent).
But still, I am unable to express the above in my yaml task. I've made quite a few guesses, an example is here:
- ${{ each project in parameters.projects }}:
- task: DotNetCoreCLI#2
displayName: Execute ${{ project }} integration tests
condition: succeededOrFailed()
inputs:
command: 'test'
projects: '$(Pipeline.Workspace)/IntegrationTests/*/*${{ project }}.Test.dll'
arguments: '--settings ${{ parameters.testRunSettingsFile }} -- TestRunParameters.Parameter(name=\"Customer1.TestConnStr\", value=\"${{ parameters.testConnStr }}\" TestRunParameters.Parameter(name=\"Customer2.TestConnStr\", value=\"${{ parameters.testConnStr }}\")'
testRunTitle: ${{ parameters.testRunTitlePrefix }}${{ project }}
In this case, the quotation marks wrapping the parameter override name and value do not appear in the generated command. Giving me an error:
The test run parameter argument is invalid. Please use the format below. Format: TestRunParameters.Parameter(name=\"<name>\", value=\"<value>\")
Substituting for properly yaml escaped single quotes as below gives me the same error:
-- TestRunParameters.Parameter(name=''Customer1.TestConnStr'', value=''${{ parameters.testConnStr }}''
So I think I am close to the solution, it may just be something with my character escaping but I just can't find how or what to escape.
Ended up solving this myself after some trial and error, just a matter of correctly escaping the literal backslash double quote. "\""
So I had the following line in my YAML:
arguments: '--settings ${{ parameters.testRunSettingsFile }} -- TestRunParameters.Parameter(name="\""Customer1.TestConnStr"\"", value="\""${{ parameters.testConnStr }}"\"" TestRunParameters.Parameter(name="\""Customer2.TestConnStr"\"", value="\""${{ parameters.testConnStr }}"\"")'
Related
I have tried to find out in what order the statements of the serverless file are evaluated (maybe it is more common to say that 'variables are resolved').
I haven't been able to find any information about this and to some extent it makes working with serverless feel like a guessing game for me.
As an example, the latest surprise I got was when I tried to run:
$ sls deploy
serverless.yaml
useDotenv: true
provider:
stage: ${env:stage}
region: ${env:region}
.env
region=us-west-1
stage=dev
I got an error message stating that env is not available at the time when stage is resolved. This was surprising to me since I have been able to use env to resolve other variables in the provider section, and there is nothing in the syntax to indicate that stage is resolved earlier.
In what order is the serverless file evaluated?
In effect you've created a circular dependency. Stage is special because it is needed to identify which .env file to load. ${env:stage} is being resolved from ${stage}.env, but Serverless needs to know what ${stage} is in order to find ${stage}.env etc.
This is why it's evaluated first.
Stage (and region, actually) are both optional CLI parameters. In your serverless.yml file what you're setting is a default, with the CLI parameter overriding it where different.
Example:
provider:
stage: staging
region: ca-central-1
Running serverless deploy --stage prod --region us-west-2 will result in prod and us-west-2 being used for stage and region (respectively) for that deployment.
I'd suggest removing any variable interpolation for stage and instead setting a default, and overriding via CLI when needed.
Then dotenv will know which environment file to use, and complete the rest of the template.
I want to run Kotlin scripts in CI without relying on a Gradle project, so I can easily do operations that would be hard to program using shell/bash/batch, and so that I can use libraries if needed.
Having the Kotlin script run only on Ubuntu/Linux is fine, though ideally, there's a way to make it run on Windows and macOS targets as well for platform specific projects.
UPDATE: Kotlin is now pre-installed on GitHub Actions runners, no need to install it beforehand anymore.
First, ensure that you have a proper Kotlin script, ending in .kts, or better, .main.kts as that latter one will be recognized better by the IDE (e.g. IntelliJ IDEA, Android Studio), especially when it comes to autocompletion and type analysis.
Second, ensure that its first line is the shebang pointing to the right place:
#!/usr/bin/env kotlin
That will be helpful to test the script locally before running in CI, as the IDE will show a run button in the gutter, next to the shebang.
If you add the execute permission to the file (chmod +x YouScript.main.kts on Linux/macOS), you'll also be able to run it just like any other script, without having to type kotlinc -script before, and that will apply on GitHub Actions as well.
Finally, here's an example manual GitHub Action (aka. workflow file) that will take an input and pass it to your Kotlin script (usable in the args property/parameter) after it installed Kotlin:
name: Run Kotlin script
on:
workflow_dispatch:
inputs:
awesome-input:
description: 'Awesome parameter'
default: 'You'
required: true
jobs:
awesome-action:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Run Kotlin script
run: kotlinc -script ./YourScript.main.kts ${{ github.event.inputs.awesome-input }}
Note that if the script has the execute (x) permission, as I told previously, you can remove the kotlinc -script part and it will still run.
Bonus: it is possible to have Kotlin code directly in the workflow file (though I'd not recommend doing it), by using kotlin as a shell.
See this YouTrack comment to see how: https://youtrack.jetbrains.com/issue/KT-43534#focus=Comments-27-4640716.0-0
Kotlin runner is now pre-installed on GitHub Actions environments (GitHub issue, YouTube video).
Refer to the GitHub Actions runner images1 to see all the installed software.
So, you can easily run your .main.kts scripts like this:
name: Example
on:
push:
branches:
- main
jobs:
example-action:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Run the script
run: kotlin /path/in/repo/to/my-script.main.kts
And here is an example my-script.main.kts file:
#file:JvmName("MyScript")
#file:CompilerOptions("-jvm-target", "11")
#file:Repository("https://repo.maven.apache.org/maven2")
// #file:DependsOn("com.example:library:1.2.3")
import java.io.File
val input = File("README.md") // Assuming you ran checkout before
val output = File("result.txt")
val readmeFirstLine = input.readLines().first()
output.writeText(readmeFirstLine)
There is also a GitHub action called setup-kotlin that lets you install your desired version of Kotlin and also provides some more features. Check out this issue.
...
- uses: actions/checkout#v3
- uses: fwilhe2/setup-kotlin#main
with:
version: 1.7.0
- name: Run the script
run: kotlin /path/in/repo/to/my-script.main.kts
It was previously called virtual environments
I'm trying to setup an auto-incrementing version number for my ASP.NET Core 3.1 web app built using Azure Pipelines. I've tried various snippets and pipeline tasks and got as far as generating the version number but my build fails with the message
The specified version string does not conform to the required format - major[.minor[.build[.revision]
I'm using the below snippet of yml to generate the version number:
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
version.MajorMinor: '1' # Manually adjust the version number as needed for semantic versioning. Revision is auto-incremented.
version.Revision: $[counter(variables['version.MajorMinor'], 0)]
versionNumber: '$(version.MajorMinor).$(version.Revision)'
steps:
- task: PowerShell#2
displayName: Set the name of the build (i.e. the Build.BuildNumber)
inputs:
targetType: 'inline'
script: |
$doy = (Get-Date).DayofYear
Write-Host "##vso[task.setvariable variable=DayOfYear]$doy"
$rev = $env:BUILD_BUILDNUMBER.Split('.')[-1]
Write-Host "##vso[task.setvariable variable=Revision]$rev
Write-Host "set Revision to $rev"
[string] $buildName = "$(versionNumber).$doy.$rev"
Write-Host "Setting the name of the build to '$buildName'."
Write-Host "##vso[build.updatebuildnumber]$buildName"
This does generate a version number (1.26.199.50), however when I get the the MSBuild task:
task: VSBuild#1
inputs:
solution: '$(solution)'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site" /p:Version=$(buildName)'
I get the version error. I've even tried replacing the version with 1.0.199.50 but I get the same error. I understand that each segment has to be less than 65534 as that's the max value for a UInt so I don't understand why it's failing.
After much frustration, I did this with a Powershell script in the release pipeline instead. I've got a Powershell task containing the below which is before the File Transform task. Steps to get it working.
Add a release pipeline variable called "MajorVersion". This is the first part of the version string
Add a property to appsettings.json called AppConfig.Version, set it to whetever for local testing.
Add a powershell task. The order should be Powershell Task->File Transform->App Service Deploy:
$doy = $(Get-Date).DayofYear
[string] $buildName = "$Env:MajorVersion.$doy.$Env:BUILD_BUILDNUMBER"
Write-Host "##vso[task.setvariable variable=AppConfig.Version;]$buildName"
Access the variable from your app settings using your preferred method. In core, probably via Configuration.GetSection("blah")
The assembly isn't stamped with the version, but you can get to it to display on the UI. I feel like this needs to be simplified by MS as auto-incrementing a version shouldn't be so difficult.
How to set gitlab-ci varibales through script not just in "varibales" section in .gitlab-ci.yaml?So that I can set variables in one job and use in different job
There is currently no way in GitLab to pass environment variable between stages or jobs.
But there is a request for that: https://gitlab.com/gitlab-org/gitlab/-/issues/22638
Current workaround is to use artifacts - basically pass files.
We had a similar use case - get Java app version from pom.xml and pass it to various jobs later in the pipeline.
How we did it in .gitlab-ci.yml:
stages:
- prepare
- package
variables:
VARIABLES_FILE: ./variables.txt # "." is required for image that have sh not bash
get-version:
stage: build
script:
- APP_VERSION=...
- echo "export APP_VERSION=$APP_VERSION" > $VARIABLES_FILE
artifacts:
paths:
- $VARIABLES_FILE
package:
stage: package
script:
- source $VARIABLES_FILE
- echo "Use env var APP_VERSION here as you like ..."
If you run a script you can set an environment variable
export MY_VAR=the-value
once the environment variable is set it should persist in the current environment.
Now for why you do not want to do that.
A tool like Gitlab CI is meant to achieve repeatability in your
artifacts. Consistency is the matter here. What happens if a second job
has to pick up a variable from the first? Then you have multiple paths!
# CI is a sequence
first -> second -> third -> fourth -> ...
# not a graph
first -> second A -> third ...
\> second B />
How did you get to third? Now if you had to debug third which path do you test? If the build in third is broken who is responsible second A or second B?
If you need a variable use it now, not later in another job/script. Whenever you
want to write a longer sequence of commands make it a script and execute the script!
You can use either Artifact or Cache to achieve this, see the official documentation for more information around Artifact and Cache:
https://docs.gitlab.com/ee/ci/caching/#how-cache-is-different-from-artifacts
I am using Azure DevOps to run some XUnit tests of an Asp.Net Core application:
- task: DotNetCoreCLI#2
displayName: 'Test'
inputs:
command: test
projects: '**/*[Tt]est/*.csproj'
arguments: '--configuration $(buildConfiguration)'
The task succeeds but I get two warnings:
[warning] Invalid results file. Make sure the result format of the file '/home/vsts/work/_temp/_fv-az592_2019-04-09_21_14_05.trx' matches 'VSTest' test results format.
[warning] Invalid results file. Make sure the result format of the file '/home/vsts/work/_temp/_fv-az592_2019-04-09_21_14_10.trx' matches 'VSTest' test results format.
What am I missing?
From github:
warning is as per our design of task since we wants to warn customer
if he is trying to use PTR task without any results, but i agree with
you about more apt warning message and we will correct it in upcoming
release.
Also try yo use VSTest task instead