Groovy - Define variable where the variable name is passed by another variable - variables

I want define a variable in groovy with where the variable name is passed by another variable.
Something like.
def runExtFunc(varName){
def varName // => def abc
varName = load 'someFile.groovy' // abc = load 'someFile.groovy'
varName."$varName"() // -> abc.abc() (run function defined in File)
}
[...]
runExtFunc('abc') // -> abc.abc() (Function abc defined in File)
[...]
runExtFunc('xyz') // -> xyz.xyz() (Function xyz defined in File)
[...]
Sadly def varName defines the variable varName and not abc. When I call runExtFunc twice an error occoures bacause varName is already defined.
I also tried
def runExtFunc(varName){
def "${varName}" // => def abc
[...]
"${varName}" = load 'someFile.groovy'
[...]
}
which doesn't work either.
Any suggestions?

This is the wrong approach. Normally you would use List, Map or Set data structures, which allow you to save a collection and access specific elements in the collection.
List allows you to hold specific values (unique or non-unique). Set allows you to hold specific values (all unique). Map allows you to have Key, Value pairs (Key must be unique) .
Read more here
groovy list,
groovy map

Try this (if I understand you correctly):
def dummyFunc(varName) {
new GroovyShell(this.binding).evaluate("${varName}")
}
dummyFunc('abc')
abc = "Hello there"
println abc
Prints
Hello there

See here
https://godless-internets.org/2020/02/14/extracting-jenkins-credentials-for-use-in-another-place/
secret_var="SECRET_VALUE_${secret_index}"
aws ssm put-parameter --name ${param_arn} --type "SecureString" --value ${!secret_var} --region us-east-2 --overwrite

I'm entering here a code sample we've done.
Please, feel free to comment.
http://groovy-lang.org/syntax.html
https://godless-internets.org/2020/02/14/extracting-jenkins-credentials-for-use-in-another-place/
def int fileContentReplaceDynamic(String filePathVar, String envTail = "",
String [] keysToIgnore = new String[0]){
def filePath = readFile filePathVar
def lines = filePath.readLines()
//def regex = ~/\b__\w*\b/
String regex = "__(.*?)__"
ArrayList credentialsList = new ArrayList();
ArrayList<String> keysToIgnoreList = new ArrayList<String>(Arrays.asList(keysToIgnore));
for (line in lines){
Pattern pattern = Pattern.compile(regex, Pattern.CASE_INSENSITIVE)
Matcher matcher = pattern.matcher(line)
while (matcher.find()){
String credKeyName = matcher.group().replaceAll("__","")
if ((! credentialsList.contains(credKeyName)) &&
(! keysToIgnoreList.contains(credKeyName))) {
credentialsList.add(credKeyName)
} else {
log.info("Credencial ignorada o ya incluida: ${credKeyName}")
}
}
}
if(credentialsList.size() <= 0){
log.info("No hay variables para aplicar transformada")
return 0
}
log.info("Numero de credenciales encontradas a sustituir: " + credentialsList.size())
String credentialsListString = String.join(", ", credentialsList);
log.info("Credenciales: " + credentialsListString)
def credsRequest = null
for(def credKeyName in credentialsList){
// Retrieve the values of the variables by environment tail name.
String credKeyNameByEnv = "${credKeyName}";
if ((envTail != null) && (! envTail.trim().isEmpty())) {
credKeyNameByEnv = credKeyNameByEnv + "-" + envTail.trim().toUpperCase();
}
// Now define the name of the variable we'll use
// List<org.jenkinsci.plugins.credentialsbinding.MultiBinding>
// Tip: java.lang.ClassCastException:
// org.jenkinsci.plugins.credentialsbinding.impl.BindingStep.bindings
// expects class org.jenkinsci.plugins.credentialsbinding.MultiBinding
String varName = "var_${credKeyNameByEnv}"
if (credsRequest == null) {
// Initialize
credsRequest = [string(credentialsId: "${credKeyNameByEnv}", variable: "${varName}")]
} else {
// Add element
credsRequest << string(credentialsId: "${credKeyNameByEnv}", variable: "${varName}")
}
}
int credsProcessed = 0
def passwordsRequest = null
StringBuilder sedReplacements = new StringBuilder();
// Now ask jenkins to fill in all variables with values
withCredentials(credsRequest) {
for(def credKeyName in credentialsList){
String credKeyVar = "var_${credKeyName}"
log.info("Replacing value for credential ${credKeyName} stored in ${credKeyVar}")
String credKeyValueIn = "${!credKeyVar}"
String credKeyValue = null;
if ("empty_string_value".equals(credKeyValueIn.trim())) {
credKeyValue = "";
} else {
credKeyValue = credKeyValueIn.replaceAll(/(!|"|#|#|\$|%|&|\/|\(|\)|=|\?)/, /\\$0/)
}
if (passwordsRequest == null) {
// Initialize
passwordsRequest = [[password: "${credKeyValue}" ]]
} else {
// Add element
passwordsRequest << [password: "${credKeyValue}" ]
}
sedReplacements.append("s/__${credKeyName}__/${credKeyValue}/; ")
credsProcessed++
}
}
wrap([$class: "MaskPasswordsBuildWrapper", varPasswordPairs: passwordsRequest ]){
String sedReplacementsString = sedReplacements.toString().trim();
if (sedReplacementsString.endsWith(";")) {
sedReplacementsString = sedReplacementsString.substring(0, sedReplacementsString.length() -1);
sedReplacementsString = sedReplacementsString + "g;"
}
sh """sed -i "${sedReplacementsString}" ${filePathVar}"""
}
log.info("Finaliza la transformada. Transformados: ${credsProcessed}/${credentialsList.size()} ")
if (credsProcessed != credentialsList.size()) {
log.info("Hay credenciales que no se han podido encontrar en el vault de Jenkins.")
log.info("Si estas guardando cadenas vacias en credenciales, guarda en su lugar el valor 'empty_string_value'.");
return -1
}
return 0;
}

Related

Kotlin, memory saving

I'm working on Splay Tree. I have a lot of command input (add, delete and etc), Hence the output is also huge (32MB of text in a txt file)
I have a memory problem, I am currently using two MultitableList and I have 200 MB of virtual memory (128 MB is needed for the project)
I run on Linux with the command:
$ kotlinc Kotlin.kt -include-runtime -d outPutFile.jar
$ /usr/bin/time -v java -jar outPutFile.jar
result: Maximum resident set size (kbytes): 249732 (With each launch different sizes but about 200)
how can i reduce the size? I need to change the size after each cycle
class SplayTree {
var ROOT: Node? = null
class Node(val key: Long, var value: String?) {
var _parent: SplayTree.Node? = null
var _RightChild: SplayTree.Node? = null
var _LeftChild: SplayTree.Node? = null
... there is a code here ...
// for output
override fun toString(): String {
if (_parent == null) {
return "[${key} ${value}]"
} else {
return "[${key} ${value} ${_parent!!.key}]"
}
}
}
... there is a code here ...
override fun toString(): String {
if (ROOT == null) {
return "_"
}
println(ROOT)
var NOWqueueList: List<Node?> = listOf(ROOT)
var BABYqueue: MutableList<Node?> = MutableList(0) { null }
//var NOWqueueList = Array<Node?>(1, {ROOT}) // Array
//var BABYqueue = Array<Node?>(1, {null}) // Array
//var n = 1 // Array
for (h in 1 until height(ROOT)) {
// for Array
//var pos = -1
//n *= 2
//BABYqueue = Array<Node?>(n, {null}) //for future line
for (ROOTList in NOWqueueList) {
//pos++ // Array
if ( ROOTList != null){
//left
if (ROOTList.haveLeftBaby()) {
//BABYqueue[pos] = ROOTList._LeftChild // Array
BABYqueue.add(ROOTList._LeftChild)
print(ROOTList._LeftChild.toString())
print(" ")
} else {
//BABYqueue[pos] = null // Array
BABYqueue.add(null)
print("_ ")
}
//pos++ // Array
//right
if (ROOTList.haveRightBaby()) {
//BABYqueue[pos] = ROOTList._RightChild // Array
BABYqueue.add(ROOTList._RightChild)
print(ROOTList._RightChild.toString())
print(" ")
} else {
//BABYqueue[pos] = null
BABYqueue.add(null)
print("_ ")
}
} else{ //если пустой то + 2 "_"
//BABYqueue[pos] = null // Array
//pos++ // Array
//BABYqueue[pos] = null // Array
BABYqueue.add(null)
BABYqueue.add(null)
print("_ _ ")
}
}
//NOWqueueList.clear() //worked when was MultitableList
NOWqueueList = BABYqueue.toList() // equate
//NOWqueueList = BABYqueue.clone() // Array
//println(NOWqueueList.joinToString(" ")) // вывожу готовый
println()
BABYqueue.clear()
}
//NOWqueueList.clear()
BABYqueue.clear()
return " end="
}
}
fun main() {
... there is a code here ...
}
I tried using Array, it still came out as with MultitableList

Using string translation in adapter return numbers

I am trying to use translation string in my adapter but it returns numbers instead
if(currentItem.budget != null){
holder.budget.text = "$ " + currentItem.budget.format()
} else {
holder.budget.text = R.string.open_to_suggestions.toString()
}
R.string.open_to_suggestions.toString() supposed to return string text Open to suggestions but it returns numbers such as 2131755113 not sure why! any idea?
To show the string resource you must use context.getString()
if(currentItem.budget != null) {
holder.budget.text = "$ " + currentItem.budget.format()
} else {
val context = holder.itemView.context
holder.budget.text = context.getString(R.string.open_to_suggestions)
}
Please take a look at the definition of getString here

what to use in place of Object.keys and hasOwnProperty when using karate version 0.9.5?

* def a =
"""
function(ths,tsa,hcia) {
karate.log(JSON.stringify(tsa[hcia[0]],null,2))
var fas = (Object.keys(tsa[hcia[0]]))[0]
karate.log("fas " + fas)
var art = null
for(prtsj in ths[hcia[0]]) {
karate.log(JSON.stringify(ths[hcia[0]][prtsj].sku,null,2))
if((Object.keys(ths[hcia[0]][prtsj].sku)).indexOf(fas) > -1) {
art = ths[hcia[0]][prtsj].type
}
}
return art
}
"""
Please use karate.keysOf() - which will return a JSON array of strings.
Now you can do whatever you want.
Also see: https://stackoverflow.com/a/61139167/143475

Map within a map in terraform variables

Does anyone know if it's possible with possibly code snipits representing whether I can create a map variable within a map variable in terraform variables?
variable "var" {
type = map
default = {
firstchoice = {
firstAChoice ="foo"
firstBChoice = "bar"
}
secondchoice = {
secondAChoice = "foobar"
secondBChoice = "barfoo"
}
}
}
If anyone has any insight to whether this is possible or any documentation that elaborates that would be great.
Yes, it's possible to have map variable as value of map variable key. Your variable just needed right indentation. Also I am putting ways to access that variable.
variable "var" {
default = {
firstchoice = {
firstAChoice = "foo"
firstBChoice = "bar"
}
secondchoice = {
secondAChoice = "foobar"
secondBChoice = "barfoo"
}
}
}
To access entire map value of a map key firstchoice, you can try following
value = "${var.var["firstchoice"]}"
output:
{
firstAChoice = foo
firstBChoice = bar
}
To access specific key of that map key (example firstAChoice), you can try
value = "${lookup(var.var["firstchoice"],"firstAChoice")}"
output: foo
Would this syntax be possible? ${var.var[firstchoice[firstAchoice]]}
With Terraform 0.12+ nested blocks are supported seamlessly. Extending #Avichal Badaya's answer to explain it using an example:
# Nested Variable
variable "test" {
default = {
firstchoice = {
firstAChoice = "foo"
firstBChoice = "bar"
}
secondchoice = {
secondAChoice = "foobar"
secondBChoice = "barfoo"
}
thirdchoice = {
thirdAChoice = {
thirdBChoice = {
thirdKey = "thirdValue"
}
}
}
}
}
# Outputs
output "firstchoice" {
value = var.test["firstchoice"]
}
output "FirstAChoice" {
value = var.test["firstchoice"]["firstAChoice"]
}
output "thirdKey" {
value = var.test["thirdchoice"]["thirdAChoice"]["thirdBChoice"]["thirdKey"]
}
Applying the above, you can verify that Terraform map nesting is now quite powerful and this makes a lot of things easier.
# Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
# Outputs:
firstchoice = {
"firstAChoice" = "foo"
"firstBChoice" = "bar"
}
thirdKey = thirdValue
For more complex structures and Rich Value Types, see HashiCorp Terraform 0.12 Preview: Rich Value Types

Delete Artifactory build artifacts using REST API

I have the following build artefacts in Artifactory server.
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.1/service_y-2.75.0.1.jar
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.2/service_y-2.75.0.2.jar
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.3/service_y-2.75.0.3.jar
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.4/service_y-2.75.0.4.jar
Questions:
I want a groovy script to delete the above artefacts except 2.75.0.3.jar (script should use Artifactory REST API). Does someone has a sample script to do that or at least delete all .jars in this case?
HOW can I use the following usage within a groovy script.
for ex: using the following line in groovy
DELETE /api/build/{buildName}[?buildNumbers=n1[,n2]][&artifacts=0/1][&deleteAll=0/1]
or
curl -X POST -v -u admin:password "http://artifactory.company.com:8081/artifactory/api/build/service_y?buildNumbers=129,130,131&artifacts=1&deleteAll=1"
Using the above mentioned curl command in Linux Putty on the same server where artifactory is installed, didn't work, gave an error.
* About to connect() to sagrdev3sb12 port 8081
* Trying 10.123.321.123... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host
http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-DeleteBuilds
or
http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-DeleteItem
The above links show their - usage sample/ usage output - confuses me.
The following link might be the answer if we can tweak this script to retain one build and delete all other builds for "projectA" (group id), "service_y" (artifact id), and for release "2.75.0.x".
https://github.com/jettro/small-scripts/blob/master/groovy/artifactory/Artifactory.groovy
I might need to use either restClient or httpBuilder within Groovy (as mentioned in the above example link and the following link).
Using Artifactory's REST API to deploy jar file
Final Answer: This scriptler script/Groovy script -- includes deleting builds from BOTH - Jenkins (using groovy it.delete()) and Artifactory (using Artifactory REST API call).
Scriptler Catalog link: http://scriptlerweb.appspot.com/script/show/103001
Enjoy!
/*** BEGIN META {
"name" : "Bulk Delete Builds except the given build number",
"comment" : "For a given job and a given build numnber, delete all builds of a given release version (M.m.interim) only and except the user provided one. Sometimes a Jenkins job use Build Name setter plugin and same job generates 2.75.0.1 and 2.76.0.43",
"parameters" : [ 'jobName', 'releaseVersion', 'buildNumber' ],
"core": "1.409",
"authors" : [
{ name : "Arun Sangal - Maddys Version" }
]
} END META **/
import groovy.json.*
import jenkins.model.*;
import hudson.model.Fingerprint.RangeSet;
import hudson.model.Job;
import hudson.model.Fingerprint;
//these should be passed in as arguments to the script
if(!artifactoryURL) throw new Exception("artifactoryURL not provided")
if(!artifactoryUser) throw new Exception("artifactoryUser not provided")
if(!artifactoryPassword) throw new Exception("artifactoryPassword not provided")
def authString = "${artifactoryUser}:${artifactoryPassword}".getBytes().encodeBase64().toString()
def artifactorySettings = [artifactoryURL: artifactoryURL, authString: authString]
if(!jobName) throw new Exception("jobName not provided")
if(!buildNumber) throw new Exception("buildNumber not provided")
def lastBuildNumber = buildNumber.toInteger() - 1;
def nextBuildNumber = buildNumber.toInteger() + 1;
def jij = jenkins.model.Jenkins.instance.getItem(jobName);
def promotedBuildRange = new Fingerprint.RangeSet()
promotedBuildRange.add(buildNumber.toInteger())
def promoteBuildsList = jij.getBuilds(promotedBuildRange)
assert promoteBuildsList.size() == 1
def promotedBuild = promoteBuildsList[0]
// The release / version of a Jenkins job - i.e. in case you use "Build name" setter plugin in Jenkins for getting builds like 2.75.0.1, 2.75.0.2, .. , 2.75.0.15 etc.
// and over the time, change the release/version value (2.75.0) to a newer value i.e. 2.75.1 or 2.76.0 and start builds of this new release/version from #1 onwards.
def releaseVersion = promotedBuild.getDisplayName().split("\\.")[0..2].join(".")
println ""
println("- Jenkins Job_Name: ${jobName} -- Version: ${releaseVersion} -- Keep Build Number: ${buildNumber}");
println ""
/** delete the indicated build and its artifacts from artifactory */
def deleteBuildFromArtifactory(String jobName, int deleteBuildNumber, Map<String, String> artifactorySettings){
println " ## Deleting >>>>>>>>>: - ${jobName}:${deleteBuildNumber} from artifactory"
def artifactSearchUri = "api/build/${jobName}?buildNumbers=${deleteBuildNumber}&artifacts=1"
def conn = "${artifactorySettings['artifactoryURL']}/${artifactSearchUri}".toURL().openConnection()
conn.setRequestProperty("Authorization", "Basic " + artifactorySettings['authString']);
conn.setRequestMethod("DELETE")
if( conn.responseCode != 200 ) {
println "Failed to delete the build artifacts from artifactory for ${jobName}/${deleteBuildNumber}: ${conn.responseCode} - ${conn.responseMessage}"
}
}
/** delete all builds in the indicated range that match the releaseVersion */
def deleteBuildsInRange(String buildRange, String releaseVersion, Job theJob, Map<String, String> artifactorySettings){
def range = RangeSet.fromString(buildRange, true);
theJob.getBuilds(range).each {
if ( it.getDisplayName().find(/${releaseVersion}.*/)) {
println " ## Deleting >>>>>>>>>: " + it.getDisplayName();
deleteBuildFromArtifactory(theJob.name, it.number, artifactorySettings)
it.delete();
}
}
}
//delete all the matching builds before the promoted build number
deleteBuildsInRange("1-${lastBuildNumber}", releaseVersion, jij, artifactorySettings)
//delete all the matching builds after the promoted build number
deleteBuildsInRange("${nextBuildNumber}-${jij.nextBuildNumber}", releaseVersion, jij, artifactorySettings)
println ""
println("- Builds have been successfully deleted for the above mentioned release: ${releaseVersion}")
println ""
I had the same question, and found this site. I took the idea and simplified it into a python script. You can find the script at: clean_artifactory.py on github
If you have any questions, please let me know. I just now cleaned up more than 10,000 snapshot artifacts!
Some of the features:
DryRun
Can dictate a time_delay, builds last_updated with in a time frame will not be removed.
Specify a targeted group such as com/foo/bar and all snapshots with in.
Hope this helps!
Wondering if the following will help:
Blog:
http://browse.feedreader.com/c/Gridshore/11546011
and
Script: https://github.com/jettro/small-scripts/blob/master/groovy/artifactory/Artifactory.groovy
package artifactory
import groovy.text.SimpleTemplateEngine
import groovyx.net.http.RESTClient
import net.sf.json.JSON
/**
* This groovy class is meant to be used to clean up your Atifactory server or get more information about it's
* contents. The api of artifactory is documented very well at the following location
* {#see http://wiki.jfrog.org/confluence/display/RTF/Artifactory%27s+REST+API}
*
* At the moment there is one major use of this class, cleaning your repository.
*
* Reading data about the repositories is done against /api/repository, if you want to remove items you need to use
* '/api/storage'
*
* Artifactory returns a strange Content Type in the response. We want to use a generic JSON library. Therefore we need
* to map the incoming type to the standard application/json. An example of the mapping is below, all the other
* mappings can be found in the obtainServerConnection method.
* 'application/vnd.org.jfrog.artifactory.storage.FolderInfo+json' => server.parser.'application/json'
*
* The class makes use of a config object. The config object is a map with a minimum of the following fields:
* def config = [
* server: 'http://localhost:8080',
* repository: 'libs-release-local',
* versionsToRemove: ['/3.2.0-build-'],
* dryRun: true]
*
* The versionsToRemove is an array of strings that are the strart of builds that should be removed. To give an idea of
* the build numbers we use: 3.2.0-build-1 or 2011.10-build-1. The -build- is important for the solution. This is how
* we identify an artifact instead of a group folder.
*
* The final option to notice is the dryRun option. This way you can get an overview of what will be deleted. If set
* to false, it will delete the selected artifacts.
*
* Usage example
* -------------
* def config = [
* server: 'http://localhost:8080',
* repository: 'libs-release-local',
* versionsToRemove: ['/3.2.0-build-'],
* dryRun: false]
*
* def artifactory = new Artifactory(config)
*
* def numberRemoved = artifactory.cleanArtifactsRecursive('nl/gridshore/toberemoved')
*
* if (config.dryRun) {* println "$numberRemoved folders would have been removed."
*} else {* println "$numberRemoved folders were removed."
*}* #author Jettro Coenradie
*/
private class Artifactory {
def engine = new SimpleTemplateEngine()
def config
def Artifactory(config) {
this.config = config
}
/**
* Print information about all the available repositories in the configured Artifactory
*/
def printRepositories() {
def server = obtainServerConnection()
def resp = server.get(path: '/artifactory/api/repositories')
if (resp.status != 200) {
println "ERROR: problem with the call: " + resp.status
System.exit(-1)
}
JSON json = resp.data
json.each {
println "key :" + it.key
println "type : " + it.type
println "descritpion : " + it.description
println "url : " + it.url
println ""
}
}
/**
* Return information about the provided path for the configured artifactory and server.
*
* #param path String representing the path to obtain information for
*
* #return JSON object containing information about the specified folder
*/
def JSON folderInfo(path) {
def binding = [repository: config.repository, path: path]
def template = engine.createTemplate('''/artifactory/api/storage/$repository/$path''').make(binding)
def query = template.toString()
def server = obtainServerConnection()
def resp = server.get(path: query)
if (resp.status != 200) {
println "ERROR: problem obtaining folder info: " + resp.status
println query
System.exit(-1)
}
return resp.data
}
/**
* Recursively removes all folders containing builds that start with the configured paths.
*
* #param path String containing the folder to check and use the childs to recursively check as well.
* #return Number with the amount of folders that were removed.
*/
def cleanArtifactsRecursive(path) {
def deleteCounter = 0
JSON json = folderInfo(path)
json.children.each {child ->
if (child.folder) {
if (isArtifactFolder(child)) {
config.versionsToRemove.each {toRemove ->
if (child.uri.startsWith(toRemove)) {
removeItem(path, child)
deleteCounter++
}
}
} else {
if (!child.uri.contains("ro-scripts")) {
deleteCounter += cleanArtifactsRecursive(path + child.uri)
}
}
}
}
return deleteCounter
}
private RESTClient obtainServerConnection() {
def server = new RESTClient(config.server)
server.parser.'application/vnd.org.jfrog.artifactory.storage.FolderInfo+json' = server.parser.'application/json'
server.parser.'application/vnd.org.jfrog.artifactory.repositories.RepositoryDetailsList+json' = server.parser.'application/json'
return server
}
private def isArtifactFolder(child) {
child.uri.contains("-build-")
}
private def removeItem(path, child) {
println "folder: " + path + child.uri + " DELETE"
def binding = [repository: config.repository, path: path + child.uri]
def template = engine.createTemplate('''/artifactory/$repository/$path''').make(binding)
def query = template.toString()
if (!config.dryRun) {
def server = new RESTClient(config.server)
server.delete(path: query)
}
}
}
Artifactory REST API would be something like (I'm not sure):
I see line: def artifactSearchUri = "api/build/${jobName}/${buildNumber}"
import groovy.json.*
def artifactoryURL= properties["jenkins.ARTIFACTORY_URL"]
def artifactoryUser = properties["artifactoryUser"]
def artifactoryPassword = properties["artifactoryPassword"]
def authString = "${artifactoryUser}:${artifactoryPassword}".getBytes().encodeBase64().toString()
def jobName = properties["jobName"]
def buildNumber = properties["buildNumber"]
def artifactSearchUri = "api/build/${jobName}/${buildNumber}"
def conn = "${artifactoryURL}/${artifactSearchUri}".toURL().openConnection()
conn.setRequestProperty("Authorization", "Basic " + authString);
println "Searching artifactory with: ${artifactSearchUri}"
def searchResults
if( conn.responseCode == 200 ) {
searchResults = new JsonSlurper().parseText(conn.content.text)
} else {
throw new Exception ("Failed to find the build info for ${jobName}/${buildNumber}: ${conn.responseCode} - ${conn.responseMessage}")
}
I had similar needs and used the script by Jettro (above) as a starting point to manage our artifacts by marking their state as a property (e.g. tested, releasable, production) and then deletes old artifacts based on the number of artifacts in each state.
The script file itself and useful adjunct files can be obtained from:
https://github.com/brianpcarr/ArtifactoryCurator
The readme is:
Invoke with:
groovy ArtifactoryProcess.groovy [--dry-run] [--full-log] --function <func> --value <val> --web-server http://YourWebServer --repository yourRepoName --domain <com/YourOrg> Version1 ...
where:
--domain domain : Name of the domain to scan.
--dry-run : Don't change anything; just list what would be done
--full-log : Log miscellaneous steps for processing artifacts
--function function : function to perform on artifacts
--maxInState maxInState : name of csv file with states and max counts, optional
--must-have mustHave : property required before applying delete, mark,
download or clear, optional
--password password : Password to use to access Artifactory server.
--repository repoName : Name of the repository to scan.
--targetDir targetDir : target directory for downloaded artifacts
--userName userName : userName to use to access Artifactory server
--value value : value to use with function above, often required
--web-server webServer : URL to use to access Artifactory server
Example: groovy ArtifactoryCleanup.groovy --domain domain --dry-run --full-log --function function --maxInState maxInState.csv --must-have mustHave --password password --repository repoName --targetDir targetDir --userName userName --value value --web-server webServer 1.0.1 1.0.2
Supported functions include [clear, delete, config, download, mark, repoPrint]
Columns in config csv files can be [repoName, targetDir, maxInState, domain, value, userName, mustHave, webServer, password, function]
The ArtifactoryProcess script can be used in a couple of main modes as well as
a sort of hyper mode.
The first mode is to mark sets of artifacts as being in a particular state, e.g.
groovy.bat ArtifactoryProcess.groovy --function mark --value production --web-server http://YourWebServer/ --repository yourRepoName --domain <com.YourOrg> --userName fill-in-userID --password fill-in-password 1.0.45-zdf
would mark all artifacts in yourRepoName with a version of 1.0.45-zdf as
being in production.
The other mode is to cleanup old artifacts. This could be done in two stages
where previously marked artifacts are deleted and then additional artifacts
would be marked for deletion on the next run.
Whether the cleanup is done in two stages or one, the different states in which
an artifact can be is defined in a comma separated value (csv) file which has
the name of the state and the number of artifacts to retain in that state. The
last entry in the MaxInState.csv file is an unnamed state and is the maximum
number of otherwise unmarked artifacts should be retained.
There is also a hyper-mode (function config) where each step of a clean up is
read from a comma separated value (csv) file. In this case the first line will
name the parameters which are to be specified and the values for each step will
be in the following lines. It is recommended that parameters like user ID and
password be passed on the command line, not in the config file.
To run the script, it can be run from the git root as suggested above. However,
this requires that groovy 2.3 or higher be installed (parameters to closure
support was added then and is required for the closure implementation used).
This requires that your JVM be at least 1.7. If you do not wish to install
groovy on your server, you can comment out the #Grapes sections at the top (oh
for the conditional compilations of c days gone) and build the required jar file
with 'gradlew build'. You can then run the utility with something like:
java -jar build/libs/artifactoryProcess-run.jar --dry-run --full-log --function mark --value tested --web-server http://YourWebServer/ --repository yourRepoName --domain <com.YourOrg> --userName fill-in-userID --password fill-in-password 1.0.45-zdf
The main script is:
package artifactoryProcess
import com.xlson.groovycsv.CsvIterator
import groovy.util.logging.Log
import org.jfrog.artifactory.client.Artifactory
import org.jfrog.artifactory.client.Repositories
import org.jfrog.artifactory.client.model.impl.RepositoryTypeImpl
import org.jfrog.artifactory.client.DownloadableArtifact;
import org.jfrog.artifactory.client.ItemHandle;
import org.jfrog.artifactory.client.PropertiesHandler;
import org.jfrog.artifactory.client.model.Folder
import org.kohsuke.args4j.Argument;
import org.kohsuke.args4j.CmdLineParser;
import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.ExampleMode;
import org.kohsuke.args4j.Option;
import groovy.transform.stc.ClosureParams;
import groovy.transform.stc.SimpleType;
import org.jfrog.artifactory.client.ArtifactoryClient;
import org.jfrog.artifactory.client.RepositoryHandle;
import com.xlson.groovycsv.CsvParser;
#Grapes([
#GrabResolver(name='jcenter', root='http://jcenter.bintray.com/', m2Compatible=true),
#Grab(group='net.sf.json-lib', module='json-lib', version='2.4', classifier='jdk15' ),
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7'),
#Grab( group='com.xlson.groovycsv', module='groovycsv', version='1.0' ),
#GrabExclude(group='org.codehaus.groovy', module='groovy-xml')
])
#Grapes( [
#Grab(group='org.kohsuke.args4j', module='args4j-maven-plugin', version='2.0.22'),
#GrabExclude(group='org.codehaus.groovy', module='groovy-xml')
])
#Grapes([
#Grab(group='org.jfrog.artifactory.client', module='artifactory-java-client-services', version='0.13'),
#GrabExclude(group='org.codehaus.groovy', module='groovy-xml')
])
/**
* This groovy class is meant to mark artifacts for release and clean up old versions.
*
* The first mode is to mark artifacts with properties such as tested, releasable and production.
*
* The second mode could be to mark artifacts for removal based on FIFO counts of artifacts in the
* states defined in the maxInState.csv file provided. If you say want at most 5 versions of an
* artifact in the production state and there are more than that, then the oldest versions could be
* marked for removal.
*
* The third mode could be the actual deletion of any artifacts which were marked for removal.
* The delay is to allow human intervention before wholesale deletion.
*
* The versionsToUse is an array of strings that are the start of builds that should be processed.
*
* There are two additional options. The first is the dryRun option. This way you can get
* an overview of what will be processed. If specified, no artifacts will be altered.
*
* Usage example
* groovy ArtifactoryProcess.groovy --dry-run --function mark --value production --must-have releasable --web-server http://yourWebServer/artifactory/ --domain <com.YourOrg> --repository libs-release-prod 1.0.1 1.0.2
*
* #author Brian Carr (snippets from Jettro Coenradie, David Carr and others)
*/
class ArtifactoryProcess {
public static final Set< String > validFunctions = [ 'mark', 'delete', 'clear', 'download', 'config', 'repoPrint' ];
public static final Set< String > validParameters = [ 'function', 'value', 'mustHave', 'targetDir', 'maxInState', 'webServer', 'repoName', 'domain', 'userName', 'password' ];
#Option(name='--dry-run', usage='Don\'t change anything; just list what would be done')
boolean dryRun;
#Option(name='--full-log', usage='Log miscellaneous steps for processing artifacts')
boolean fullLog;
// eg --function mark
#Option(name='--function', metaVar='function', usage="function to perform on artifacts")
String function;
// eg --value production
#Option(name='--value', metaVar='value', usage="value to use with function above, often required")
String value;
// eg --must-have releasable
#Option(name='--must-have', metaVar='mustHave', usage="property required before applying delete, mark, download or clear, optional")
String mustHave;
// eg --targetDir d:/temp/bin
#Option(name='--targetDir', metaVar='targetDir', usage="target directory for downloaded artifacts")
String targetDir;
// eg --maxInState MaxInState.csv
#Option(name='--maxInState', metaVar='maxInState', usage="name of csv file with states and max counts, optional")
String maxInState;
// eg --web-server 'http://artifactory01/artifactory/'
#Option(name='--web-server', metaVar='webServer', usage='URL to use to access Artifactory server')
String webServer;
// eg --repository 'libs-release-prod'
#Option(name='--repository', metaVar='repoName', usage='Name of the repository to scan.')
String repoName;
// eg --domain 'org/apache'
#Option(name='--domain', metaVar='domain', usage='Name of the domain to scan.')
String domain;
// eg --userName cleaner
#Option(name='--userName', metaVar='userName', usage='userName to use to access Artifactory server')
String userName;
// eg --password SomePswd
#Option(name='--password', metaVar='password', usage='Password to use to access Artifactory server.')
String password;
#Argument
ArrayList<String> versionsToUse = new ArrayList<String>();
class PathAndDate{
String path;
Date dtCreated;
}
class StateRecord {
String state;
int cnt;
List< PathAndDate > pathAndDate;
}
#SuppressWarnings(["SystemExit", "CatchThrowable"])
static void main( String[] args ) {
try {
new ArtifactoryProcess().doMain( args );
} catch (Throwable throwable) {
// Java returns exit code 0 if it terminates because of an uncaught Throwable.
// That's bad if we have a process like Bamboo depending on errors being non-zero.
// Thus, we catch all Throwables and explicitly set the exit code.
println( "Unexpected error: ${throwable}" )
System.exit(1)
}
System.exit(0);
}
List< StateRecord > stateSet = [];
Artifactory srvr;
RepositoryHandle repo;
private int numProcessed = 0;
String firstFunction;
String lastConfig;
void doMain( String[] args ) {
CmdLineParser parser = new CmdLineParser( this );
try {
parser.parseArgument(args);
if( function == 'config' && value == null ) {
throw new CmdLineException("You must provide a config.csv file as the value if you specify the config function.");
}
firstFunction = function; // Flag in case we recurse into config files, where did we start
if( function == 'config' ) {
processConfig();
return;
} else {
checkParms();
}
} catch(CmdLineException ex) {
System.err.println(ex.getMessage());
System.err.println();
System.err.println("groovy ArtifactoryProcess.groovy [--dry-run] [--full-log] --function <func> --value <val> --web-server http://YourWebServer --repository libs-release-prod --domain <com/YourOrg> Version1 ...");
parser.printUsage(System.err);
System.err.println();
System.err.println(" Example: groovy ArtifactoryProcess.groovy"+parser.printExample(ExampleMode.ALL)+" 1.0.1 1.0.2");
System.err.println();
System.err.println(" Supported functions include ${validFunctions}" );
System.err.println();
System.err.println(" Columns in config csv files can be ${validParameters}" );
return;
}
String stateLims;
if( maxInState != null && maxInState.size() > 0 ) stateLims = "(using stateLims)" else stateLims = "(no stateLims)"
println( "Started processing of $function with ${(value==null)?mustHave:value} $stateLims on $webServer in $repoName/$domain with $versionsToUse." );
withClient { newClient ->
srvr = newClient;
if( function == 'repoPrint' ) printRepositories();
else {
processRepo();
}
}
}
def processRepo() {
numProcessed = 0; // Reset count from last repo.
repo = srvr.repository( repoName );
processArtifactsRecursive( domain );
if( dryRun ) {
println "$numProcessed folders would have been $function[ed] with $value.";
} else {
println "$numProcessed folders were $function[ed] with $value.";
}
}
def processConfig() {
File configCSV = new File( value );
lastConfig = value; // Record which csv file we have last dived into.
Artifactory mySrvr = srvr; // Each line of config could have a different web server, preserve connection in case recursing
configCSV.withReader {
CsvIterator csvIt = CsvParser.parseCsv( it );
for( csvRec in csvIt ) {
if (fullLog) println("Step is ${csvRec}");
Map cols = csvRec.properties.columns;
String func = csvRec.function;
def hasFunc = cols.containsKey( 'function' );
def has = cols.containsKey( 'targetDir' );
if( cols.containsKey( 'function' ) && !noValue( csvRec.function ) ) function = csvRec.function ;
if( cols.containsKey( 'value' ) && !noValue( csvRec.value ) ) value = csvRec.value ;
if( cols.containsKey( 'targetDir' ) && !noValue( csvRec.targetDir ) ) targetDir = csvRec.targetDir ;
if( cols.containsKey( 'maxInState' ) && !noValue( csvRec.maxInState ) ) maxInState = csvRec.maxInState;
if( cols.containsKey( 'webServer' ) && !noValue( csvRec.webServer ) ) webServer = csvRec.webServer ;
if( cols.containsKey( 'repoName' ) && !noValue( csvRec.repoName ) ) repoName = csvRec.repoName ;
if( cols.containsKey( 'domain' ) && !noValue( csvRec.domain ) ) repoName = csvRec.domain ;
if( cols.containsKey( 'userName' ) && !noValue( csvRec.userName ) ) userName = csvRec.userName ;
if( cols.containsKey( 'password' ) && !noValue( csvRec.password ) ) password = csvRec.password ;
if( cols.containsKey( 'mustHave' ) ) mustHave = csvRec.mustHave; // Can clear out mustHave value
checkParms();
withClient { newClient ->
srvr = newClient;
processRepo();
}
srvr = mySrvr; // Restore previous web server connection
}
}
}
def checkParms() {
if( !noValue( maxInState ) ) {
stateSet.clear(); // Throw away any previous states from last step
File stateFile = new File( maxInState );
def RC = stateFile.withReader {
CsvIterator csvFile = CsvParser.parseCsv( it );
for( csvRec in csvFile ) {
String state = csvRec.properties.values[ 0 ];
String strCnt = csvRec.properties.values[ 1 ];
if( fullLog ) println( "State ${state} allowed ${strCnt}" );
int count = 0;
if( strCnt.integer ) count = strCnt.toInteger();
if( count < 0 ) count = 0;
// Iterator lies and claims there is a next when there isn't. Force break on empty state.
stateSet.add( new StateRecord( state: state, cnt: count, pathAndDate: [] ) );
}
}
}
String prefix;
if( firstFunction == 'config' && function != 'config' ) {
prefix = "While processing ${lastConfig} encountered, ";
} else prefix = '';
if( !validFunctions.contains( function ) ) {
throw new CmdLineException( "${prefix}Unrecognized function ${function}, function is required and must be one of ${validFunctions}." );
}
if( function == 'mark' && noValue( value ) ) {
throw new CmdLineException( "${prefix}You must provide a value to mark with if you specify the mark function." );
}
if( function == 'clear' && noValue( value ) ) {
throw new CmdLineException( "${prefix}You must provide a value to clear with if you specify the clear function." );
}
if( function != 'repoPrint' && noValue( domain ) ) {
throw new CmdLineException( "${prefix}You must provide a domain to use with the ${function} function." );
}
if( function == 'download' ) {
if( noValue( targetDir ) ) targetDir = '.';
}
if( noValue( webServer ) || noValue( userName ) || noValue( password ) || noValue( repoName ) ) {
throw new CmdLineException( "${prefix}You must provide the webServer, userName, password and repository name values to use." );
}
if( versionsToUse.size() == 0 && stateSet.size() == 0 && function != 'repoPrint' ) {
throw new CmdLineException( "${prefix}You must provide maxInState or a list of artifacts / versions to act upon." );
}
}
Boolean noValue( var ) {
return var == null || var == '';
}
/**
* Print information about all the available repositories in the configured Artifactory
*/
def printRepositories() {
Repositories repos = srvr.repositories();
List repoList = repos.list( RepositoryTypeImpl.LOCAL );
for( it in repoList ) {
println "key :" + it.key
println "type : " + it.type
println "description : " + it.description
println "url : " + it.url
println ""
};
}
/**
* Recursively removes all folders containing builds that start with the configured paths.
*
* #param path String containing the folder to check and use the childs to recursively check as well.
* #return Number with the amount of folders that were processed.
*/
private int processArtifactsRecursive( String path ) {
ItemHandle item = repo.folder( path );
// def RC = item.isFolder(); This lies, always returns true even for a file, go figure!
// def RC = path.endsWith('.xml'); // item.info() fails for simple files, go figure!
if( !path.endsWith( '.xml' ) &&
!path.endsWith( '.jar' ) &&
item.isFolder() ) {
Folder fldr;
try{
fldr = item.info()
} catch( Exception e ) {
println( "Error accessing $webServer/$repoName/$path" );
throw( e );
};
for( kid in fldr.children ) {
boolean processed = false;
if( stateSet.size() > 0 ) {
if( isEndNode( kid.uri )) {
processed = groupFolders( path + kid.uri );
}
} else {
versionsToUse.find { version ->
if( kid.uri.startsWith( '/' + version ) ) {
numProcessed += processItem( path + kid.uri );
return true; // Once we find a match, no others are interesting, we are outta here
} else return false; // Just formalize the on to next iterator
}
}
if( !processed ) {
processArtifactsRecursive( path + kid.uri );
}
}
}
/* If we are counting number in each state, our lists should be all set now */
if( stateSet.size() > 0 ) {
processSet();
}
return numProcessed;
}
private boolean processedThis( String vrsn, kid ) {
if( kid.uri.startsWith('/' + vrsn )) {
numProcessed += processItem( vrsn + kid.uri );
return true; // Once we find a match, no others are interesting, we are outta here
} else return false; // Just formalize the on to next iterator
}
// True if nodeName is of form int.int.other, could be one line, but how would you debug it.
private boolean isEndNode( String nodeName ){
int firstDot = nodeName.indexOf( '.' );
if( firstDot <= 1 ) return false; // nodeName starts with '/' which is ignored
int secondDot = nodeName.indexOf( '.', firstDot + 1 );
if( secondDot <= 0 ) return false;
String firstInt = nodeName.substring( 1, firstDot ); // nodeName starts with '/' which is ignored
if( !firstInt.isInteger() ) return false;
String secondInt = nodeName.substring( firstDot + 1, secondDot );
if( secondInt.isInteger() ) return true;
return false;
}
private boolean groupFolders( String path ) {
Map<String, List<String>> props;
stateSet.find { rec ->
ItemHandle folder = repo.folder( path );
if( rec.state.size() > 0 ) {
props = folder.getProperties( rec.state );
}
if( rec.state.size() <= 0 || props.size() > 0 ) {
PathAndDate nodePathDate = new PathAndDate();
nodePathDate.path = path;
nodePathDate.dtCreated = folder.info().lastModified;
rec.pathAndDate.add( nodePathDate ); // process this one
return true; // No others are interest, break out of iterator
} else return false; // On to next iterator
}
return true; // We always process all nodes which are end nodes
}
private boolean processSet() {
for( set in stateSet ) {
int del = set.cnt;
if( set.pathAndDate.size() < del ) {
del = set.pathAndDate.size() }
else {
set.pathAndDate.sort() { a,b -> b.dtCreated <=> a.dtCreated }; // Sort newest first to preserve newest
}
while( del > 0 ) {
set.pathAndDate.remove( 0 );
del--;
}
while( set.pathAndDate.size() > 0 ) {
numProcessed += processItem( set.pathAndDate[ 0 ].path );
set.pathAndDate.remove( 0 );
}
}
return true;
}
private int processItem( String path ) {
int retVal = 0;
if( fullLog ) println "Processing folder: ${path}, ${function} with ${value}.";
def RC;
ItemHandle folder = repo.folder( path );
Map<String, List<String>> props;
boolean hasRqrd = true;
if( !noValue( mustHave ) ) {
props = folder.getProperties( mustHave );
if( props.size() > 0 ) hasRqrd = true; else hasRqrd = false; // I like this better than ternary operator, you?
}
if( !hasRqrd ) return retVal;
switch( function ) {
case 'delete':
if( !dryRun ) RC = repo.delete( path );
retVal++;
break;
case 'download':
if( folder.isFolder() ) {
Folder item = folder.info();
item.children.find() { kid ->
if( kid.uri.endsWith('.jar') ) {
DownloadableArtifact DA = repo.download( path + kid.uri );
InputStream dlJar = DA.doDownload(); // Open Source
FileWriter lclJar = new FileWriter( targetDir + kid.uri, false ); // Open Dest
for( id in dlJar ) { lclJar.write( id ); } // Copy contents
if( fullLog ) println( "Downloaded ${path + kid.uri} to ${targetDir + kid.uri}." );
retVal++;
return true;
}
}
}
break;
case 'mark':
props = folder.getProperties( value );
if( props.size() == 0 ) {
PropertiesHandler item = folder.properties();
PropertiesHandler PH = item.addProperty( value, 'true' );
if( !dryRun ) RC = PH.doSet();
retVal++;
}
break;
case 'clear':
props = folder.getProperties( value );
if( props.size() == 1 ) {
if( !dryRun ) RC = folder.deleteProperty( value ); // Null return is success, go figure!
retVal++;
}
break;
default:
println( "Unknown function $function with $value encountered on ${path}.")
}
if( retVal > 0 ) println "Completed $function on $path with ${(value==null)?mustHave:value}.";
return retVal;
}
private <T> T withClient( #ClosureParams( value = SimpleType, options = "org.jfrog.artifactory.client.Artifactory" ) Closure<T> closure ) {
def client = ArtifactoryClient.create( "${webServer}artifactory", userName, password )
try {
return closure( client )
} finally {
client.close()
}
}
}