I am running a local version of the NoFlo Development Environment and would like to know how to remove (unregister) a runtime. Actually, how can I remove a runtime from the FlowHub hosted environment, as well?
There is currently no UI to do this, but the API exists: Issue
Here is my bash script for doing just that.
#!/bin/bash -x
# Your UUID can be found through developer JS console: Resources -> Local Storage -> Look for grid-token
uuid="<your uuid>"
# the list of runtimes you want to delete.
list=$1
for i in ${list}
do
curl -X DELETE http://api.flowhub.io/runtimes/${i} -H "Authorization: Bearer ${uuid}"
done
Related
Context
Our current build system builds docker images inside of a docker container (Docker in Docker). Many of our docker builds need credentials to be able to pull from private artifact repositories.
We've handled this with docker secrets.. passing in the secret to the docker build command, and in the Dockerfile, referencing the secret in the RUN command where its needed. This means we're using docker buildkit. This article explains it.
We are moving to a different build system (GitLab) and the admins have disabled Docker in Docker (security reasons) so we are moving to Kaniko for docker builds.
Problem
Kaniko doesn't appear to support secrets the way docker does. (there are no command line options to pass a secret through the Kaniko executor).
The credentials the docker build needs are stored in GitLab variables. For DinD, you simply add those variables to the docker build as a secret:
DOCKER_BUILDKIT=1 docker build . \
--secret=type=env,id=USERNAME \
--secret=type=env,id=PASSWORD \
And then in docker, use the secret:
RUN --mount=type=secret,id=USERNAME --mount=type=secret,id=PASSWORD \
USER=$(cat /run/secrets/USERNAME) \
PASS=$(cat /run/secrets/PASSWORD) \
./scriptThatUsesTheseEnvVarCredentialsToPullArtifacts
...rest of build..
Without the --secret flag to the kaniko executor, I'm not sure how to take advantage of docker secrets... nor do I understand the alternatives. I also want to continue to support developer builds. We have a 'build.sh' script that takes care of gathering credentials and adding them to the docker build command.
Current Solution
I found this article and was able to sort out a working solution. I want to ask the experts if this is valid or what the alternatives might be.
I discovered that when the kaniko executor runs, it appears to mount a volume into the image that's being built at: /kaniko. That directory does not exist when the build is complete and does not appear to be cached in the docker layers.
I also found out that if if the Dockerfile secret is not passed in via the docker build command, the build still executes.
So my gitlab-ci.yml file has this excerpt (the REPO_USER/REPO_PWD variables are GitLab CI variables):
- echo "${REPO_USER}" > /kaniko/repo-credentials.txt
- echo "${REPO_PWD}" >> /kaniko/repo-credentials.txt
- /kaniko/executor
--context "${CI_PROJECT_DIR}/docker/target"
--dockerfile "${CI_PROJECT_DIR}/docker/target/Dockerfile"
--destination "${IMAGE_NAME}:${BUILD_TAG}"
Key piece here is echo'ing the credentials to a file in the /kaniko directory before calling the executor. That directory is (temporarily) mounted into the image which the executor is building. And since all this happens inside of the kaniko image, that file will disappear when kaniko (gitlab) job completes.
The developer build script (snip):
//to keep it simple, this assumes that the developer has their credentials//cached in a file (ignored by git) called dev-credentials.txt
DOCKER_BUILDKIT=1 docker build . \
--secret id=repo-creds,src=dev-credentials.txt
Basically same as before. Had to put it in a file instead of environment variables.
The dockerfile (snip):
RUN --mount=type=secret,id=repo-creds,target=/kaniko/repo-credentials.txt USER=$(sed '1q;d' /kaniko/repo-credentials.txt) PASS=$(sed '2q;d' /kaniko/repo-credentials.txt) ./scriptThatUsesTheseEnvVarCredentialsToPullArtifacts...rest of build..
This Works!
In the Dockerfile, by mounting the secret in the /kaniko subfolder, it will work with both the DinD developer build as well as with the CI Kaniko executor.
For Dev builds, DinD secret works as always. (had to change it to a file rather than env variables which I didn't love.)
When the build is run by Kaniko, I suppose since the secret in the RUN command is not found, it doesn't even try to write the temporary credentials file (which I expected would fail the build). Instead, because I directly wrote the varibles to the temporarily mounted /kaniko directory, the rest of the run command was happy.
Advice
To me this does seem more kludgy than expected. I'm wanting to find out other/alternative solutions. Finding out the /kaniko folder is mounted into the image at build time seems to open a lot of possibilities.
I'm trying to sort the list of artifacts from jfrog artifactory but getting (The requested URL returned error: 400 Bad Request), in the jfrog documentation (https://www.jfrog.com/confluence/display/JFROG/Artifactory+Comparison+Matrix) says it won't work for open source services. After we get list of artifacts need to delete old artifacts from subfolder in the artifactory repo. Tried with CLI and AQL but nothing worked.
Our repo url looks like this
http://domainname/artifactory/repo/folder/subfolder/test1.zip
Like test 1.zip we have many artifacts(let's say 50)in that subfolder. Looking for help on this, anyone pls me on this issue. Thanks.
While sorting is not supported in OSS versions, if you would like to delete artifacts older than a certain time period, you can use Relative Time Operators, parse the output, and use a script to delete those artifacts.
You can also specify a specific date. There are several Comparison Operators that you can use.
You can use the below AQL for reference:
curl -uadmin:password -XPOST "http://localhost:8082/artifactory/api/search/aql" -d 'items.find({"repo": "repo"}, {"path": "folder/subfolder"}, {"created" : {"$before" : "2minutes"}})' -H "Content-Type: text/plain"
This is command for packaging edge extension -
manifoldjs -l debug -p edgeextension -f edgeextension -m \manifest.json.
- I gave here the path of my extension manifest.json file.
After manifoldjs finished running following folders should create -
My Extension
edgeextension
generationInfo.json
manifest
AppXManifest.xml
Assets
Logo.png
Extension
manifest.json
popup.html
...
But in my case, folders are not creating and I am unable to package my extension.
I try to check the description and it looks like you are developing an extension for the MS Edge legacy browser.
I try to refer to the official docs and try to create and package sample extension.
I got the output below:
When I try to check the folder then I noticed that folder is empty. So based on my test result I am able to produce the issue.
I try to check the old threads and found that it is a known issue.
I found that issue occurs if the users are using the Node V10. In that thread, users had mentioned that it will work fine with the Node V8. I suggest trying to make a test with the Node V8 may help you to fix the issue.
Further, I suggest you can try to provide feedback about this issue on their Github page.
I'm writing a script to automatically setup a repository starting from a clean GraphDB running in a Docker container.
I have a config.ttl file containing repository configuration, the namespace and a dump in a file init.nq
I have successfully created the repository using the config.ttf and updated namespace but I cannot understand how to load the init.nq file.
This operation is extremely simple from web interface: Import -> RFD -> Upload, but I'm not able to understand how to perform it using Curl. I suppose that the correct API should be
post /repositories/{repositoryID}/statements
but the dump is to huge to pass it as simple text (~44MB).
This should work:
curl -X POST -H "Content-Type:application/n-quads" -T init.nq 'http://localhost:7200/repositories/test/statements'
I am working to set up a gitlab runner for multiple projects, and we want to be able to set up environment variables for all of the projects. I tried to set global variables in the .bashrc for both the gitlab-runner and root users but it did not recognize them during the CI script. What is the correct location to declare global environment variables?
You can also inject environment variables to your gitlab-runner directly in the commandline, as the gitlab-runner exec docker --help states:
OPTIONS: ..
--env value Custom environment variables injected to build environment [$RUNNER_ENV] ..
Here is a small example how I use it in a script:
Change the declarations as needed:
declare jobname="your_jobname"
declare runnerdir="/path/to/your/repository"
Get the env file into a bash array.
[ -f "$runnerdir/env" ] \
&& declare -a envlines=($(cat "$runnerdir/env"))
declare -a envs=()
for env in "${envlines[#]}"; do
envs+=(--env "$env")
done
And finally pass it to the gitlab-runner.
[ -d "$runnerdir" ] && cd "$runnerdir" \
&& gitlab-runner exec docker "${envs[#]}" $jobname \
&& cd -
You can define environment variables to inject in the runner's config.toml file. See the advanced runner configuration documentation in the [[runners]] section.
There doesn't seem to be a way to specify environment variables in the GitLab UI just for a specific runner.
With GitLab 13.1 (June 2020), you now have:
Instance-level CI/CD variables
GitLab now supports instance-level variables.
With this ability to set global variables, you no longer need to manually enter the same credentials repeatedly for all your projects.
This MVC introduces access to this feature by API, and the next iteration of this feature will provide the ability to configure instance-level variables directly in the UI.
See Documentation and issue.
Consider using an external persistent secret storage service like Vault or Keywhiz
Disclaimer: I am not associated nor used any of the above services
I have added export MY_VAR="FOO" to gitlab-runner's .bashrc, and it works.
echo export MY_VAR=\"FOO\" >> /home/gitlab-runner/.bashrc
Check which type of executor do you use? (shell, kubernetes, docker-ssh, parallels...) I use shell executor.
Check what type of shell does gitlab-runner use? (How to determine the current shell I'm working on) And edit the proper rc file for that.
Check the Gitlab CI Runner user.
I suggest dump all environment variables for further debugging, by add env to the .gitlab-ci.yml's script:
#.gitlab-ci.yml
job:
script: env
Making changes in ~/.bash_profile NOT ~/.bashrc.
See my answer
You can easily setup Variables in the GitLab Settings:
Project-level variables can be added by going to your project's Settings > CI/CD, then finding the section called Variables.
To make sure your variables are only used in
See:
https://docs.gitlab.com/ee/ci/variables/#variables