Related
I have this code:
class kg is Dimension {
method new() {
return self.bless(
:type('mass'),
:abbr('kg'),
:multiplier(Multiplier.new(
numerator => 1.0,
denominator => Quantity.new(1000.0, 'g')))),
}
}
class mg is Dimension {
method new() {
return self.bless(
:type('mass'),
:abbr('mg'),
:multiplier(Multiplier.new(
numerator => 1000.0,
denominator => Quantity.new(1.0, 'g')))),
}
}
I'll be adding many more similar classes. Rather than spell out all these classes separately, I'd like to learn how to create a factory that can create these classes from simple data structures.
How do I do this? I read the Metaobject Protocol doc but I couldn't figure out how to give my classes different names on the fly based on the examples at the top and middle of the doc page.
I tried:
constant A := Metamodel::ClassHOW.new_type( name => 'A' );
A.^add_method('x', my method x(A:) { say 42 });
A.^add_method('set', my method set(A: Mu \a) { A.^set_name(a) });
A.^compose;
my $bar = A;
$bar.set('Foo');
say $bar.^name; #
A.x; # works
Foo.x; # error
But the last line just throws an error:
Undeclared name:
Foo used at line 13
The first thing you should realize, that any kind of meta-programmming usually will need to be done at compile time, aka in a BEGIN block.
Secondly: at the moment, Raku has some meta-programming features for creating code, but not all features needed to make this as painless as possible. The work on RakuAST will change that, as it basically makes Raku itself being built from a public meta-programming API (rather than you could argue, the current bootstrap version using a lot of NQP).
I've rewritten your code to the following:
sub frobnicate(Str:D $name) {
my \A := Metamodel::ClassHOW.new_type(:$name);
A.^add_method('x', my method x() { say 42 });
A.^compose;
OUR::{$name} := A;
}
BEGIN frobnicate("Foo");
say Foo.^name; # Foo
Foo.x; # 42
So, this introduces a sub called frobnicate that creates a new type with the given name. Adds a method x to it, and composes the new type. And then makes sure it is known as an our in the current compilation unit.
Then the frobnicate sub is called at compile time by prefixing a BEGIN. This is important, because otherwise Foo won't be known when the next line is compiled, so you'd get errors.
There is currently a small catch:
dd Foo.^find_method("x").signature; # :(Mu: *%_)
The invocant constraint is not set. I haven't found a way (before RakuAST) to set that using an meta-programming interface. But I don't think that's going to be an issue for the example you've given. If it does become an issue, then let's cross that bridge when we get there.
Here is the entire code that I came up with for a solution:
#!/usr/bin/env raku
use v6.d;
class Dimension { }
sub dimension-attr-factory($name, Mu $type, Mu $package) {
return Attribute.new(
:name('$.' ~ $name),
:type($type),
:has_accessor(1),
#:is_required(1),
:package($package)
);
}
sub dimension-factory(Str:D $name, #attributes) {
my \A := Metamodel::ClassHOW.new_type(:$name);
A.^add_parent(Dimension);
for #attributes {
my $attr = dimension-attr-factory($_[0], $_[1], A);
A.^add_attribute($attr);
}
A.^compose;
OUR::{$name} := A;
}
class Multiplier {
has Rat $.numerator;
has Quantity $.denominator;
method factor() {
return $.numerator / $.denominator.value;
}
}
class Quantity {
has Rat() $.value is required;
has Dimension:D $.dimension is required;
multi submethod BUILD(Rat:D() :$!value, Dimension:D :$!dimension) {
}
multi submethod BUILD(Rat:D() :$value, Str:D :$dimension) {
$!dimension = ::($dimension).new;
}
multi method new(Rat:D() $value, Dimension:D $dimension) {
return self.bless(
:$value,
:$dimension,
)
}
multi method new(Rat:D() $value, Str:D $dimension) {
return self.bless(
:$value,
:$dimension,
)
}
method to(Str:D $dimension = '') {
my $from_value = $.value;
my $to = $dimension ?? ::($dimension).new !! ::(self.dimension.abbr).new;
# do types match?
if $to.type ne self.dimension.type {
die "Cannot convert a " ~ self.dimension.type ~ " to a " ~ $to.type;
};
my $divisor = $.dimension.multiplier ?? $.dimension.multiplier.factor !! 1.0;
my $dividend = $to.multiplier ?? $to.multiplier.factor !! 1;
my $result = $dividend / $divisor * $from_value;
return Quantity.new($result, $to);
}
method gist() {
$.value ~ ' ' ~ $.dimension.abbr;
}
}
BEGIN {
my %dimensions = 'mass' => {
base => {
abbr => 'g',
},
derived => {
kg => { num => 1000.0, den => 1.0, },
mg => { num => 1.0, den => 1000.0, },
ug => { num => 1.0, den => 1000000.0, },
}
};
for %dimensions.kv -> $key, $value {
# set up base class for dimension type
my $base = %dimensions{$key}<base><abbr>;
my #attributes = ['abbr', $base], ['type', $key];
dimension-factory( $base, #attributes);
my %derived = %dimensions{$key}<derived>;
for %derived.kv -> $abbr, $values {
my $numerator = %dimensions{$key}<derived>{$abbr}<num>;
my $denominator = %dimensions{$key}<derived>{$abbr}<den>;
my $multiplier = Multiplier.new(
numerator => 1.0,
denominator => Quantity.new(1000.0, 'g'),
);
#attributes = ['abbr', $abbr], ['type', $key], ['multiplier', $multiplier];
my $dim = dimension-factory( $abbr, #attributes );
#$dim.new(:$abbr, type => $key, :$multiplier );
}
}
}
my $kg = kg.new();
my $quant = Quantity.new(value => 5.0, dimension => $kg);
dd $quant;
I would probably create a dimension keyword with a custom metamodel, would probably also override * and / operators using undefined dimensions and then would create kg with something like:
dimension Gram {
has Dimension::Type $.type = mass;
has Str $.abbr = "g";
}
dimension KiloGram is Gram {
has Str $.abbr = "kg";
has Dimension::Multiplier $.multiplier = 1000 * g;
}
dimension MiliGram is Gram {
has Str $.abbr = "mg";
has Dimension::Multiplier $.multiplier = g / 1000;
}
but maybe that's too much...
Please consider the code below. Why is the output of this is "BABABA" and not "AAABAA" / "AABAAAB"? Shouldn't the two supplies run in parallel and the whenever fire immedeatly when there is an event in any of them?
my $i = 0;
my $supply1 = supply { loop { await Promise.in(3); done if $i++> 5; emit("B"); } };
my $supply2 = supply { loop { await Promise.in(1); done if $i++> 5; emit("A"); } };
react
{
#whenever Supply.merge($supply1, $supply2) -> $x { $x.print }
whenever $supply1 -> $x { $x.print };
whenever $supply2 -> $x { $x.print };
}
When we subscribe to a supply block, the body of that supply block is run immediately in order to set up subscriptions. There's no concurrency introduced as part of this; if we want that, we need to ask for it.
The best solution depends on how close the example is to what you're doing. If it's very close - and you want to emit values every time interval - then the solution is to use Supply.interval instead:
my $i = 0;
my $supply1 = supply { whenever Supply.interval(3, 3) { done if $i++ > 5; emit("B"); } };
my $supply2 = supply { whenever Supply.interval(1, 1) { done if $i++> 5; emit("A"); } };
react {
whenever $supply1 -> $x { $x.print };
whenever $supply2 -> $x { $x.print };
}
Which simply sets up a subscription and gets out of the setup, and so gives the output you want, however you do have a data race on the $i.
The more general pattern is to just do anything that gets the loop happening out of the setup step. For example, we could use an a kept Promise to just "thunk" it:
my constant READY = Promise.kept;
my $i = 0;
my $supply1 = supply whenever READY {
loop { await Promise.in(3); done if $i++> 5; emit("B"); }
}
my $supply2 = supply whenever READY {
loop { await Promise.in(1); done if $i++> 5; emit("A"); }
}
react {
whenever $supply1 -> $x { $x.print };
whenever $supply2 -> $x { $x.print };
}
Which helps because the result of a Promise will be delivered to the supply block via the thread pool scheduler, thus forcing the execution of the content of the whenever - containing the loop - into its own scheduled task.
This isn't especially pretty, but if we define a function to do it:
sub asynchronize(Supply $s) {
supply whenever Promise.kept {
whenever $s { .emit }
}
}
Then the original program only needs the addition of two calls to it:
my $i = 0;
my $supply1 = supply { loop { await Promise.in(3); done if $i++> 5; emit("B") } }
my $supply2 = supply { loop { await Promise.in(1); done if $i++> 5; emit("A") } }
react {
whenever asynchronize $supply1 -> $x { $x.print }
whenever asynchronize $supply2 -> $x { $x.print }
}
To make it work as desired. Arguably, something like this should be provided as a built-in.
It is possible to use a Channel as well, as the other solution proposes, and depending on the problem at hand that may be suitable; the question is a bit too abstracted from a real problem for me to say. This solution stays within the Supply paradigm, and is neater in that sense.
Thanks to jjmerelo here, I managed to get it working. The channel was the right track, but you actually have to consume the channels supply.
use v6;
my Channel $c .= new;
my $supply1 = start { loop { await Promise.in(1); $c.send("B"); } };
my $supply2 = start { loop { await Promise.in(0.5); $c.send("A"); } };
react
{
whenever $c.Supply -> $x { $x.print };
}
$c.close;
Additional question: How good does that scale? Can you have several thousand supplies sending to the channel?
Supplies are asynchronous, not concurrent. You will need to use channels instead of supplies to feed them concurrently.
use v6;
my $i = 0;
my Channel $c .= new;
my $supply1 = start { for ^5 { await Promise.in(1); $c.send("B"); } };
my $supply2 = start { for ^5 { await Promise.in(0.5); $c.send("A"); } };
await $supply2;
await $supply1;
$c.close;
.say for $c.list;
In this case, the two threads start at the same time, and instead of using .emit, then .send to the channel. In your example, they are effectively blocked while they wait, since they are both running in the same thread. They only give control to the other supply after the promise is kept, so that they run apparently "in parallel" and as slow as the slower of them.
Ok, so here is my real code. It seems to work, but I think there is a race condition somewhere. Here's some typical (albeit short) output.
A monster hatched.
A monster hatched.
A hero was born.
The Monster is at 2,3
The Monster is at 3,2
The Player is at 0,0
The Monster (2) attacks the Player (3)
The Monster rolls 14
The Player rolls 4
The Monster inflicts 4 damage
The Player (3) attacks the Monster (2)
The Player rolls 11
The Monster rolls 8
The Player inflicts 45 damage
The Monster is dead
The Monster is at -3,-3
The Player is at 4,-3
The Monster (1) attacks the Player (3)
The Monster rolls 8
The Player rolls 5
The Monster inflicts 11 damage
The Player has 32 hitpoints left
The Monster is at -4,1
The Player is at -1,4
The Player (3) attacks the Monster (1)
The Player rolls 12
The Monster rolls 11
The Player inflicts 46 damage
The Monster is dead
Stopping
Game over. The Player has won
Now the strange thing is, sometimes, in maybe 20% of the runs, the last line of the output is
Game over. The GameObject has won
as if the object got caught while it already is partially deconstructed? Or something? Anyway here's the code.
class GameObject
{
has Int $.id;
has Int $.x is rw;
has Int $.y is rw;
has $.game;
has Int $.speed; #the higher the faster
has Bool $.stopped is rw;
multi method start( &action )
{
start {
loop {
&action();
last if self.stopped;
await Promise.in( 1 / self.speed );
}
$.game.remove-object( self );
}
}
method speed {
$!speed +
# 33% variation from the base speed in either direction
( -($!speed / 3).Int .. ($!speed / 3).Int ).pick
;
}
}
role UnnecessaryViolence
{
has $.damage;
has $.hitpoints is rw;
has $.offense;
has $.defense;
method attack ( GameObject $target )
{
say "The {self.WHAT.perl} ({self.id}) attacks the {$target.WHAT.perl} ({$target.id})";
my $attacker = roll( $.offense, 1 .. 6 ).sum;
say "The {self.WHAT.perl} rolls $attacker";
my $defender = roll( $target.defense, 1 .. 6 ).sum;
say "The {$target.WHAT.perl} rolls $defender";
if $attacker > $defender
{
my $damage = ( 1 .. $.damage ).pick;
say "The {self.WHAT.perl} inflicts {$damage} damage";
$target.hitpoints -= $damage ;
}
if $target.hitpoints < 0
{
say "The {$target.WHAT.perl} is dead";
$target.stopped = True;
}
else
{
say "The {$target.WHAT.perl} has { $target.hitpoints } hitpoints left";
}
}
}
class Player is GameObject does UnnecessaryViolence
{
has $.name;
multi method start
{
say "A hero was born.";
self.start({
# say "The hero is moving";
# keyboard logic here, in the meantime random movement
$.game.channel.send( { object => self, x => (-1 .. 1).pick, y => (-1 .. 1).pick } );
});
}
}
class Monster is GameObject does UnnecessaryViolence
{
has $.species;
multi method start
{
say "A monster hatched.";
self.start({
# say "The monster {self.id} is moving";
# AI logic here, in the meantime random movement
$.game.channel.send( { object => self, x => (-1 .. 1).pick, y => (-1 .. 1).pick } );
});
}
}
class Game
{
my $idc = 0;
has GameObject #.objects is rw;
has Channel $.channel = .new;
method run{
self.setup;
self.mainloop;
}
method setup
{
self.add-object( Monster.new( :id(++$idc), :species("Troll"), :hitpoints(20), :damage(14), :offense(3), :speed(300), :defense(3), :x(3), :y(2), :game(self) ) );
self.add-object( Monster.new( :id(++$idc), :species("Troll"), :hitpoints(10), :damage(16), :offense(3), :speed(400), :defense(3), :x(3), :y(2), :game(self) ) );
self.add-object( Player.new( :id(++$idc), :name("Holli"), :hitpoints(50), :damage(60), :offense(3), :speed(200) :defense(2), :x(0), :y(0), :game(self) ) );
}
method add-object( GameObject $object )
{
#!objects.push( $object );
$object.start;
}
method remove-object( GameObject $object )
{
#!objects = #!objects.grep({ !($_ === $object) });
}
method mainloop
{
react {
whenever $.channel.Supply -> $event
{
self.stop-game
if self.all-objects-stopped;
self.process-movement( $event );
self.stop-objects
if self.game-is-over;
};
whenever Supply.interval(1) {
self.render;
}
}
}
method process-movement( $event )
{
#say "The {$event<object>.WHAT.perl} moves.";
given $event<object>
{
my $to-x = .x + $event<x>;
my $to-y = .y + $event<y>;
for #!objects -> $object
{
# we don't care abour ourselves
next
if $_ === $object;
# see if anything is where we want to be
if ( $to-x == $object.x && $to-y == $object.y )
{
# can't move, blocked by friendly
return
if $object.WHAT eqv .WHAT;
# we found a monster
.attack( $object );
last;
}
}
# -5 -1 5
# we won the fight or the place is empty
# so let's move
.x = $to-x == 5 ?? -4 !!
$to-x == -5 ?? 4 !!
$to-x;
.y = $to-y == 5 ?? -4 !!
$to-y == -5 ?? 4 !!
$to-y;
}
}
method render
{
for #!objects -> $object {
"The {$object.WHAT.perl} is at {$object.x},{$object.y}".say;
}
}
method stop-objects
{
say "Stopping";
for #!objects -> $object {
$object.stopped = True;
}
}
method stop-game {
"Game over. The {#!objects[0].WHAT.perl} has won".say;
$.channel.close;
done;
}
method game-is-over {
return (#!objects.map({.WHAT})).unique.elems == 1;
}
method all-objects-stopped {
(#!objects.grep({!.stopped})).elems == 0;
}
}
Game.new.run;
There is an issue with the translations, if the translation is missing prestashop is returining empty string, rather than the key.
Does anyone know the location of the 'l' method used in the controllers?
$this->l('string', 'mod'); //This will output '' if string is not translated.
I want to modify the method and make it return the key if the value is empty, but I cant find it.
I'll assume you are referring to an AdminController, since it's the only one using that function. It uses the function:
protected function l($string, $class = null, $addslashes = false, $htmlentities = true)
{
if ($class === null || $class == 'AdminTab') {
$class = substr(get_class($this), 0, -10);
} elseif (strtolower(substr($class, -10)) == 'controller') {
/* classname has changed, from AdminXXX to AdminXXXController, so we remove 10 characters and we keep same keys */
$class = substr($class, 0, -10);
}
return Translate::getAdminTranslation($string, $class, $addslashes, $htmlentities);
}
In your case it would call Translate::getAdminTranslation('string', 'mod', false, true)
In Translate::getAdminTranslation
We have:
...
$string = preg_replace("/\\\*'/", "\'", $string);
$key = md5($string);
if (isset($_LANGADM[$class.$key])) {
$str = $_LANGADM[$class.$key];
} else {
$str = Translate::getGenericAdminTranslation($string, $key, $_LANGADM);
}
...
Since it won't have the $_LANGADM[$class.$key], it will call:
$str = Translate::getGenericAdminTranslation($string, $key, $_LANGADM);
in your case $str = Translate::getGenericAdminTranslation('string', md5('string'), $_LANGADM);
In there we have:
...
if (isset($lang_array['AdminController'.$key])) {
$str = $lang_array['AdminController'.$key];
} elseif (isset($lang_array['Helper'.$key])) {
$str = $lang_array['Helper'.$key];
} elseif (isset($lang_array['AdminTab'.$key])) {
$str = $lang_array['AdminTab'.$key];
} else {
// note in 1.5, some translations has moved from AdminXX to helper/*.tpl
$str = $string;
}
return $str;
So by default if no key is found, the same string that is trying to be translated is returned. So there is no need to change the function.
On the otherhand, make sure the string it's translated to an empty string. You can also debug these functions to make sure your class is correct, and the file that is storing the corresponding translations doesn't have the empty translation for those strings.
Here's exercise 5.F.2 from 'A Book of Abstract Algebra' by Charles C Pinter:
Let G be the group {e, a, b, b^2, b^3, ab, ab^2, ab^3} whose
generators satisfy a^2 = e, b^4 = e, ba = ab^3. Write the table
of G. (G is called the dihedral group D4.)
Here's a little Perl 6 program which presents a solution:
sub generate(%eqs, $s)
{
my #results = ();
for %eqs.kv -> $key, $val {
if $s ~~ /$key/ { #results.push($s.subst(/$key/, $val)); }
if $s ~~ /$val/ { #results.push($s.subst(/$val/, $key)); }
}
for #results -> $result { take $result; }
my #arrs = #results.map({ gather generate(%eqs, $_) });
my $i = 0;
while (1)
{
for #arrs -> #arr { take #arr[$i]; }
$i++;
}
}
sub table(#G, %eqs)
{
printf " |"; for #G -> $y { printf "%-5s|", $y; }; say '';
printf "-----|"; for #G -> $y { printf "-----|"; }; say '';
for #G -> $x {
printf "%-5s|", $x;
for #G -> $y {
my $result = (gather generate(%eqs, "$x$y")).first(* (elem) #G);
printf "%-5s|", $result;
}
say ''
}
}
# ----------------------------------------------------------------------
# Pinter 5.F.2
my #G = <e a b bb bbb ab abb abbb>;
my %eqs = <aa e bbbb e ba abbb>; %eqs<e> = '';
table #G, %eqs;
Here's what the resulting table looks like:
Let's focus on these particular lines from generate:
my #arrs = #results.map({ gather generate(%eqs, $_) });
my $i = 0;
while (1)
{
for #arrs -> #arr { take #arr[$i]; }
$i++;
}
A recursive call to generate is made for each of the items in #results. Then we're effectively performing a manual 'zip' on the resulting sequences. However, Perl 6 has zip and the Z operator.
Instead of the above lines, I'd like to do something like this:
for ([Z] #results.map({ gather generate(%eqs, $_) })).flat -> $elt { take $elt; }
So here's the full generate using Z:
sub generate(%eqs, $s)
{
my #results = ();
for %eqs.kv -> $key, $val {
if $s ~~ /$key/ { #results.push($s.subst(/$key/, $val)); }
if $s ~~ /$val/ { #results.push($s.subst(/$val/, $key)); }
}
for #results -> $result { take $result; }
for ([Z] #results.map({ gather generate(%eqs, $_) })).flat -> $elt { take $elt; }
}
The issue with the Z version of generate is that it hangs...
So, my question is, is there a way to write generate in terms of Z?
Besides this core question, feel free to share alternative solutions to the exercise which explore and showcase Perl 6.
As another example, here's exercise 5.F.3 from the same book:
Let G be the group {e, a, b, b^2, b^3, ab, ab^2, ab^3} whose
generators satisfy a^4 = e, a^2 = b^2, ba = ab^3. Write the
table of G. (G is called the quaternion group.)
And the program above displaying the table:
As an aside, this program was converted from a version in C#. Here's how generate looks there using LINQ and a version of ZipMany courtesy of Eric Lippert.
static IEnumerable<string> generate(Dictionary<string,string> eqs, string s)
{
var results = new List<string>();
foreach (var elt in eqs)
{
if (new Regex(elt.Key).IsMatch(s))
results.Add(new Regex(elt.Key).Replace(s, elt.Value, 1));
if (new Regex(elt.Value).IsMatch(s))
results.Add(new Regex(elt.Value).Replace(s, elt.Key, 1));
}
foreach (var result in results) yield return result;
foreach (var elt in ZipMany(results.Select(elt => generate(eqs, elt)), elts => elts).SelectMany(elts => elts))
yield return elt;
}
The entire C# program: link.
[2022 update by #raiph. I just tested the first block of code in a recent Rakudo. The fourth example returned one result, 'abc', rather than none. This may be due to a new Raku design decision / roast improvement / trap introduced since this answer was last edited (in 2017), or a Rakudo bug. I'm not going to investigate; I just wanted to let readers know.]
Why your use of zip doesn't work
Your code assumes that [Z] ("reducing with the zip operator") can be used to get the transpose of a list-of-lists.
Unfortunately, this doesn't work in the general case.
It 'usually' works, but breaks on one edge case: Namely, when the list-of-lists is a list of exactly one list. Observe:
my #a = <a b c>, <1 2 3>, <X Y Z>; put [Z~] #a; # a1X b2Y c3Z
my #a = <a b c>, <1 2 3>; put [Z~] #a; # a1 b2 c3
my #a = <a b c>,; put [Z~] #a; # abc
my #a; put [Z~] #a; # abc <-- 2022 update
In the first two examples (3 and 2 sub-lists), you can see that the transpose of #a was returned just fine. The fourth example (0 sub-lists) does the right thing as well.
But the third example (1 sub-list) didn't print a b c as one would expect, i.e. it didn't return the transpose of #a in that case, but rather (it seems) the transpose of #a[0].
Sadly, this is not a Rakudo bug (in which case it could simply be fixed), but an unforseen interaction of two Perl 6 design decisions, namely:
The reduce meta-operator [ ] handles an input list with a single element by calling the operator it's applied to with one argument (said element).
In case you're wondering, an infix operator can be called with only one argument by invoking its function object: &infix:<Z>( <a b c>, ).
The zip operator Z and function zip (like other built-ins that accept nested lists), follows the so-called "single-argument rule" – i.e. its signature uses a single-argument slurpy parameter. This means that when it is called with a single argument, it will descend into it and consider its elements the actual arguments to use. (See also Slurpy conventions.)
So zip(<a b c>,) is treated as zip("a", "b", "c").
Both features provide some nice convenience in many other cases, but in this case their interaction regrettably poses a trap.
How to make it work with zip
You could check the number of elements of #arrs, and special-case the "exactly 1 sub-list" case:
my #arrs = #results.map({ gather generate(%eqs, $_) });
if #arrs.elems == 1 {
.take for #arrs[0][];
}
else {
.take for flat [Z] #arrs
}
The [] is a "zen slice" - it returns the list unchanged, but without the item container that the parent Array wrapped it in. This is needed because the for loop would consider anything wrapped in an item container as a single item and only do one iteration.
Of course, this if-else solution is not very elegant, which probably negates your reason for trying to use zip in the first place.
How to write the code more elegantly without zip
Refer to Christoph's answer.
It might be possible with a Z, but for my poor little brain, zipping recursively generated lazy lists is too much.
Instead, I did some other simplifications:
sub generate($s, %eqs) {
take $s;
# the given equations normalize the string, ie there's no need to apply
# the inverse relation
for %eqs.kv -> $k, $v {
# make copy of $s so we can use s/// instead of .subst
my $t = $s;
generate $t, %eqs
if $t ~~ s/$k/$v/;
}
}
sub table(#G, %eqs) {
# compute the set only once instead of implicitly on each call to (elem)
my $G = set #G;
# some code golfing
put ['', |#G]>>.fmt('%-5s|').join;
put '-----|' x #G + 1;
for #G -> $x {
printf '%-5s|', $x;
for #G -> $y {
printf '%-5s|', (gather generate("$x$y", %eqs)).first(* (elem) $G);
}
put '';
}
}
my #G = <e a b bb bbb ab abb abbb>;
# use double brackets so we can have empty strings
my %eqs = <<aa e bbbb e ba abbb e ''>>;
table #G, %eqs;
Here is a compact rewrite of generate that does bidirectional substitution, still without an explicit zip:
sub generate($s, %eqs) {
my #results = do for |%eqs.pairs, |%eqs.antipairs -> (:$key, :$value) {
take $s.subst($key, $value) if $s ~~ /$key/;
}
my #seqs = #results.map: { gather generate($_, %eqs) }
for 0..* -> $i { take .[$i] for #seqs }
}
Here's a version of generate that uses the approach demonstrated by smls:
sub generate(%eqs, $s)
{
my #results = ();
for %eqs.kv -> $key, $val {
if $s ~~ /$key/ { #results.push($s.subst(/$key/, $val)); }
if $s ~~ /$val/ { #results.push($s.subst(/$val/, $key)); }
}
for #results -> $result { take $result; }
my #arrs = #results.map({ gather generate(%eqs, $_) });
if #arrs.elems == 1 { .take for #arrs[0][]; }
else { .take for flat [Z] #arrs; }
}
I've tested it and it works on exercises 2 and 3.
As smls mentions in his answer, zip doesn't do what we were expecting when the given array of arrays only contains a single array. So, let's make a version of zip which does work with one or more arrays:
sub zip-many (#arrs)
{
if #arrs.elems == 1 { .take for #arrs[0][]; }
else { .take for flat [Z] #arrs; }
}
And now, generate in terms of zip-many:
sub generate(%eqs, $s)
{
my #results = ();
for %eqs.kv -> $key, $val {
if $s ~~ /$key/ { #results.push($s.subst(/$key/, $val)); }
if $s ~~ /$val/ { #results.push($s.subst(/$val/, $key)); }
}
for #results -> $result { take $result; }
zip-many #results.map({ gather generate(%eqs, $_) });
}
That looks pretty good.
Thanks smls!
smls suggests in a comment below that zip-many not invoke take, leaving that to generate. Let's also move flat from zip-many to generate.
The slimmed down zip-many:
sub zip-many (#arrs) { #arrs == 1 ?? #arrs[0][] !! [Z] #arrs }
And the generate to go along with it:
sub generate(%eqs, $s)
{
my #results;
for %eqs.kv -> $key, $val {
if $s ~~ /$key/ { #results.push($s.subst(/$key/, $val)); }
if $s ~~ /$val/ { #results.push($s.subst(/$val/, $key)); }
}
.take for #results;
.take for flat zip-many #results.map({ gather generate(%eqs, $_) });
}
Testing the keys and values separately seems a bit silly; your strings aren't really regexes, so there's no need for // anywhere in your code.
sub generate($s, #eqs) {
my #results = do for #eqs.kv -> $i, $equation {
take $s.subst($equation, #eqs[ $i +^ 1 ]) if $s.index: $equation
}
my #seqs = #results.map: { gather generate($_, #eqs) }
for 0..* -> $i { take .[$i] for #seqs }
}
Obviously with this version of generate you'll have to rewrite table to use #eqs instead of %eqs.
I have the following build artefacts in Artifactory server.
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.1/service_y-2.75.0.1.jar
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.2/service_y-2.75.0.2.jar
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.3/service_y-2.75.0.3.jar
http://artifactory.company.com:8081/artifactory/libs-snapshot-local/com/mycompany/projectA/service_y/2.75.0.4/service_y-2.75.0.4.jar
Questions:
I want a groovy script to delete the above artefacts except 2.75.0.3.jar (script should use Artifactory REST API). Does someone has a sample script to do that or at least delete all .jars in this case?
HOW can I use the following usage within a groovy script.
for ex: using the following line in groovy
DELETE /api/build/{buildName}[?buildNumbers=n1[,n2]][&artifacts=0/1][&deleteAll=0/1]
or
curl -X POST -v -u admin:password "http://artifactory.company.com:8081/artifactory/api/build/service_y?buildNumbers=129,130,131&artifacts=1&deleteAll=1"
Using the above mentioned curl command in Linux Putty on the same server where artifactory is installed, didn't work, gave an error.
* About to connect() to sagrdev3sb12 port 8081
* Trying 10.123.321.123... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host
http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-DeleteBuilds
or
http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-DeleteItem
The above links show their - usage sample/ usage output - confuses me.
The following link might be the answer if we can tweak this script to retain one build and delete all other builds for "projectA" (group id), "service_y" (artifact id), and for release "2.75.0.x".
https://github.com/jettro/small-scripts/blob/master/groovy/artifactory/Artifactory.groovy
I might need to use either restClient or httpBuilder within Groovy (as mentioned in the above example link and the following link).
Using Artifactory's REST API to deploy jar file
Final Answer: This scriptler script/Groovy script -- includes deleting builds from BOTH - Jenkins (using groovy it.delete()) and Artifactory (using Artifactory REST API call).
Scriptler Catalog link: http://scriptlerweb.appspot.com/script/show/103001
Enjoy!
/*** BEGIN META {
"name" : "Bulk Delete Builds except the given build number",
"comment" : "For a given job and a given build numnber, delete all builds of a given release version (M.m.interim) only and except the user provided one. Sometimes a Jenkins job use Build Name setter plugin and same job generates 2.75.0.1 and 2.76.0.43",
"parameters" : [ 'jobName', 'releaseVersion', 'buildNumber' ],
"core": "1.409",
"authors" : [
{ name : "Arun Sangal - Maddys Version" }
]
} END META **/
import groovy.json.*
import jenkins.model.*;
import hudson.model.Fingerprint.RangeSet;
import hudson.model.Job;
import hudson.model.Fingerprint;
//these should be passed in as arguments to the script
if(!artifactoryURL) throw new Exception("artifactoryURL not provided")
if(!artifactoryUser) throw new Exception("artifactoryUser not provided")
if(!artifactoryPassword) throw new Exception("artifactoryPassword not provided")
def authString = "${artifactoryUser}:${artifactoryPassword}".getBytes().encodeBase64().toString()
def artifactorySettings = [artifactoryURL: artifactoryURL, authString: authString]
if(!jobName) throw new Exception("jobName not provided")
if(!buildNumber) throw new Exception("buildNumber not provided")
def lastBuildNumber = buildNumber.toInteger() - 1;
def nextBuildNumber = buildNumber.toInteger() + 1;
def jij = jenkins.model.Jenkins.instance.getItem(jobName);
def promotedBuildRange = new Fingerprint.RangeSet()
promotedBuildRange.add(buildNumber.toInteger())
def promoteBuildsList = jij.getBuilds(promotedBuildRange)
assert promoteBuildsList.size() == 1
def promotedBuild = promoteBuildsList[0]
// The release / version of a Jenkins job - i.e. in case you use "Build name" setter plugin in Jenkins for getting builds like 2.75.0.1, 2.75.0.2, .. , 2.75.0.15 etc.
// and over the time, change the release/version value (2.75.0) to a newer value i.e. 2.75.1 or 2.76.0 and start builds of this new release/version from #1 onwards.
def releaseVersion = promotedBuild.getDisplayName().split("\\.")[0..2].join(".")
println ""
println("- Jenkins Job_Name: ${jobName} -- Version: ${releaseVersion} -- Keep Build Number: ${buildNumber}");
println ""
/** delete the indicated build and its artifacts from artifactory */
def deleteBuildFromArtifactory(String jobName, int deleteBuildNumber, Map<String, String> artifactorySettings){
println " ## Deleting >>>>>>>>>: - ${jobName}:${deleteBuildNumber} from artifactory"
def artifactSearchUri = "api/build/${jobName}?buildNumbers=${deleteBuildNumber}&artifacts=1"
def conn = "${artifactorySettings['artifactoryURL']}/${artifactSearchUri}".toURL().openConnection()
conn.setRequestProperty("Authorization", "Basic " + artifactorySettings['authString']);
conn.setRequestMethod("DELETE")
if( conn.responseCode != 200 ) {
println "Failed to delete the build artifacts from artifactory for ${jobName}/${deleteBuildNumber}: ${conn.responseCode} - ${conn.responseMessage}"
}
}
/** delete all builds in the indicated range that match the releaseVersion */
def deleteBuildsInRange(String buildRange, String releaseVersion, Job theJob, Map<String, String> artifactorySettings){
def range = RangeSet.fromString(buildRange, true);
theJob.getBuilds(range).each {
if ( it.getDisplayName().find(/${releaseVersion}.*/)) {
println " ## Deleting >>>>>>>>>: " + it.getDisplayName();
deleteBuildFromArtifactory(theJob.name, it.number, artifactorySettings)
it.delete();
}
}
}
//delete all the matching builds before the promoted build number
deleteBuildsInRange("1-${lastBuildNumber}", releaseVersion, jij, artifactorySettings)
//delete all the matching builds after the promoted build number
deleteBuildsInRange("${nextBuildNumber}-${jij.nextBuildNumber}", releaseVersion, jij, artifactorySettings)
println ""
println("- Builds have been successfully deleted for the above mentioned release: ${releaseVersion}")
println ""
I had the same question, and found this site. I took the idea and simplified it into a python script. You can find the script at: clean_artifactory.py on github
If you have any questions, please let me know. I just now cleaned up more than 10,000 snapshot artifacts!
Some of the features:
DryRun
Can dictate a time_delay, builds last_updated with in a time frame will not be removed.
Specify a targeted group such as com/foo/bar and all snapshots with in.
Hope this helps!
Wondering if the following will help:
Blog:
http://browse.feedreader.com/c/Gridshore/11546011
and
Script: https://github.com/jettro/small-scripts/blob/master/groovy/artifactory/Artifactory.groovy
package artifactory
import groovy.text.SimpleTemplateEngine
import groovyx.net.http.RESTClient
import net.sf.json.JSON
/**
* This groovy class is meant to be used to clean up your Atifactory server or get more information about it's
* contents. The api of artifactory is documented very well at the following location
* {#see http://wiki.jfrog.org/confluence/display/RTF/Artifactory%27s+REST+API}
*
* At the moment there is one major use of this class, cleaning your repository.
*
* Reading data about the repositories is done against /api/repository, if you want to remove items you need to use
* '/api/storage'
*
* Artifactory returns a strange Content Type in the response. We want to use a generic JSON library. Therefore we need
* to map the incoming type to the standard application/json. An example of the mapping is below, all the other
* mappings can be found in the obtainServerConnection method.
* 'application/vnd.org.jfrog.artifactory.storage.FolderInfo+json' => server.parser.'application/json'
*
* The class makes use of a config object. The config object is a map with a minimum of the following fields:
* def config = [
* server: 'http://localhost:8080',
* repository: 'libs-release-local',
* versionsToRemove: ['/3.2.0-build-'],
* dryRun: true]
*
* The versionsToRemove is an array of strings that are the strart of builds that should be removed. To give an idea of
* the build numbers we use: 3.2.0-build-1 or 2011.10-build-1. The -build- is important for the solution. This is how
* we identify an artifact instead of a group folder.
*
* The final option to notice is the dryRun option. This way you can get an overview of what will be deleted. If set
* to false, it will delete the selected artifacts.
*
* Usage example
* -------------
* def config = [
* server: 'http://localhost:8080',
* repository: 'libs-release-local',
* versionsToRemove: ['/3.2.0-build-'],
* dryRun: false]
*
* def artifactory = new Artifactory(config)
*
* def numberRemoved = artifactory.cleanArtifactsRecursive('nl/gridshore/toberemoved')
*
* if (config.dryRun) {* println "$numberRemoved folders would have been removed."
*} else {* println "$numberRemoved folders were removed."
*}* #author Jettro Coenradie
*/
private class Artifactory {
def engine = new SimpleTemplateEngine()
def config
def Artifactory(config) {
this.config = config
}
/**
* Print information about all the available repositories in the configured Artifactory
*/
def printRepositories() {
def server = obtainServerConnection()
def resp = server.get(path: '/artifactory/api/repositories')
if (resp.status != 200) {
println "ERROR: problem with the call: " + resp.status
System.exit(-1)
}
JSON json = resp.data
json.each {
println "key :" + it.key
println "type : " + it.type
println "descritpion : " + it.description
println "url : " + it.url
println ""
}
}
/**
* Return information about the provided path for the configured artifactory and server.
*
* #param path String representing the path to obtain information for
*
* #return JSON object containing information about the specified folder
*/
def JSON folderInfo(path) {
def binding = [repository: config.repository, path: path]
def template = engine.createTemplate('''/artifactory/api/storage/$repository/$path''').make(binding)
def query = template.toString()
def server = obtainServerConnection()
def resp = server.get(path: query)
if (resp.status != 200) {
println "ERROR: problem obtaining folder info: " + resp.status
println query
System.exit(-1)
}
return resp.data
}
/**
* Recursively removes all folders containing builds that start with the configured paths.
*
* #param path String containing the folder to check and use the childs to recursively check as well.
* #return Number with the amount of folders that were removed.
*/
def cleanArtifactsRecursive(path) {
def deleteCounter = 0
JSON json = folderInfo(path)
json.children.each {child ->
if (child.folder) {
if (isArtifactFolder(child)) {
config.versionsToRemove.each {toRemove ->
if (child.uri.startsWith(toRemove)) {
removeItem(path, child)
deleteCounter++
}
}
} else {
if (!child.uri.contains("ro-scripts")) {
deleteCounter += cleanArtifactsRecursive(path + child.uri)
}
}
}
}
return deleteCounter
}
private RESTClient obtainServerConnection() {
def server = new RESTClient(config.server)
server.parser.'application/vnd.org.jfrog.artifactory.storage.FolderInfo+json' = server.parser.'application/json'
server.parser.'application/vnd.org.jfrog.artifactory.repositories.RepositoryDetailsList+json' = server.parser.'application/json'
return server
}
private def isArtifactFolder(child) {
child.uri.contains("-build-")
}
private def removeItem(path, child) {
println "folder: " + path + child.uri + " DELETE"
def binding = [repository: config.repository, path: path + child.uri]
def template = engine.createTemplate('''/artifactory/$repository/$path''').make(binding)
def query = template.toString()
if (!config.dryRun) {
def server = new RESTClient(config.server)
server.delete(path: query)
}
}
}
Artifactory REST API would be something like (I'm not sure):
I see line: def artifactSearchUri = "api/build/${jobName}/${buildNumber}"
import groovy.json.*
def artifactoryURL= properties["jenkins.ARTIFACTORY_URL"]
def artifactoryUser = properties["artifactoryUser"]
def artifactoryPassword = properties["artifactoryPassword"]
def authString = "${artifactoryUser}:${artifactoryPassword}".getBytes().encodeBase64().toString()
def jobName = properties["jobName"]
def buildNumber = properties["buildNumber"]
def artifactSearchUri = "api/build/${jobName}/${buildNumber}"
def conn = "${artifactoryURL}/${artifactSearchUri}".toURL().openConnection()
conn.setRequestProperty("Authorization", "Basic " + authString);
println "Searching artifactory with: ${artifactSearchUri}"
def searchResults
if( conn.responseCode == 200 ) {
searchResults = new JsonSlurper().parseText(conn.content.text)
} else {
throw new Exception ("Failed to find the build info for ${jobName}/${buildNumber}: ${conn.responseCode} - ${conn.responseMessage}")
}
I had similar needs and used the script by Jettro (above) as a starting point to manage our artifacts by marking their state as a property (e.g. tested, releasable, production) and then deletes old artifacts based on the number of artifacts in each state.
The script file itself and useful adjunct files can be obtained from:
https://github.com/brianpcarr/ArtifactoryCurator
The readme is:
Invoke with:
groovy ArtifactoryProcess.groovy [--dry-run] [--full-log] --function <func> --value <val> --web-server http://YourWebServer --repository yourRepoName --domain <com/YourOrg> Version1 ...
where:
--domain domain : Name of the domain to scan.
--dry-run : Don't change anything; just list what would be done
--full-log : Log miscellaneous steps for processing artifacts
--function function : function to perform on artifacts
--maxInState maxInState : name of csv file with states and max counts, optional
--must-have mustHave : property required before applying delete, mark,
download or clear, optional
--password password : Password to use to access Artifactory server.
--repository repoName : Name of the repository to scan.
--targetDir targetDir : target directory for downloaded artifacts
--userName userName : userName to use to access Artifactory server
--value value : value to use with function above, often required
--web-server webServer : URL to use to access Artifactory server
Example: groovy ArtifactoryCleanup.groovy --domain domain --dry-run --full-log --function function --maxInState maxInState.csv --must-have mustHave --password password --repository repoName --targetDir targetDir --userName userName --value value --web-server webServer 1.0.1 1.0.2
Supported functions include [clear, delete, config, download, mark, repoPrint]
Columns in config csv files can be [repoName, targetDir, maxInState, domain, value, userName, mustHave, webServer, password, function]
The ArtifactoryProcess script can be used in a couple of main modes as well as
a sort of hyper mode.
The first mode is to mark sets of artifacts as being in a particular state, e.g.
groovy.bat ArtifactoryProcess.groovy --function mark --value production --web-server http://YourWebServer/ --repository yourRepoName --domain <com.YourOrg> --userName fill-in-userID --password fill-in-password 1.0.45-zdf
would mark all artifacts in yourRepoName with a version of 1.0.45-zdf as
being in production.
The other mode is to cleanup old artifacts. This could be done in two stages
where previously marked artifacts are deleted and then additional artifacts
would be marked for deletion on the next run.
Whether the cleanup is done in two stages or one, the different states in which
an artifact can be is defined in a comma separated value (csv) file which has
the name of the state and the number of artifacts to retain in that state. The
last entry in the MaxInState.csv file is an unnamed state and is the maximum
number of otherwise unmarked artifacts should be retained.
There is also a hyper-mode (function config) where each step of a clean up is
read from a comma separated value (csv) file. In this case the first line will
name the parameters which are to be specified and the values for each step will
be in the following lines. It is recommended that parameters like user ID and
password be passed on the command line, not in the config file.
To run the script, it can be run from the git root as suggested above. However,
this requires that groovy 2.3 or higher be installed (parameters to closure
support was added then and is required for the closure implementation used).
This requires that your JVM be at least 1.7. If you do not wish to install
groovy on your server, you can comment out the #Grapes sections at the top (oh
for the conditional compilations of c days gone) and build the required jar file
with 'gradlew build'. You can then run the utility with something like:
java -jar build/libs/artifactoryProcess-run.jar --dry-run --full-log --function mark --value tested --web-server http://YourWebServer/ --repository yourRepoName --domain <com.YourOrg> --userName fill-in-userID --password fill-in-password 1.0.45-zdf
The main script is:
package artifactoryProcess
import com.xlson.groovycsv.CsvIterator
import groovy.util.logging.Log
import org.jfrog.artifactory.client.Artifactory
import org.jfrog.artifactory.client.Repositories
import org.jfrog.artifactory.client.model.impl.RepositoryTypeImpl
import org.jfrog.artifactory.client.DownloadableArtifact;
import org.jfrog.artifactory.client.ItemHandle;
import org.jfrog.artifactory.client.PropertiesHandler;
import org.jfrog.artifactory.client.model.Folder
import org.kohsuke.args4j.Argument;
import org.kohsuke.args4j.CmdLineParser;
import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.ExampleMode;
import org.kohsuke.args4j.Option;
import groovy.transform.stc.ClosureParams;
import groovy.transform.stc.SimpleType;
import org.jfrog.artifactory.client.ArtifactoryClient;
import org.jfrog.artifactory.client.RepositoryHandle;
import com.xlson.groovycsv.CsvParser;
#Grapes([
#GrabResolver(name='jcenter', root='http://jcenter.bintray.com/', m2Compatible=true),
#Grab(group='net.sf.json-lib', module='json-lib', version='2.4', classifier='jdk15' ),
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7'),
#Grab( group='com.xlson.groovycsv', module='groovycsv', version='1.0' ),
#GrabExclude(group='org.codehaus.groovy', module='groovy-xml')
])
#Grapes( [
#Grab(group='org.kohsuke.args4j', module='args4j-maven-plugin', version='2.0.22'),
#GrabExclude(group='org.codehaus.groovy', module='groovy-xml')
])
#Grapes([
#Grab(group='org.jfrog.artifactory.client', module='artifactory-java-client-services', version='0.13'),
#GrabExclude(group='org.codehaus.groovy', module='groovy-xml')
])
/**
* This groovy class is meant to mark artifacts for release and clean up old versions.
*
* The first mode is to mark artifacts with properties such as tested, releasable and production.
*
* The second mode could be to mark artifacts for removal based on FIFO counts of artifacts in the
* states defined in the maxInState.csv file provided. If you say want at most 5 versions of an
* artifact in the production state and there are more than that, then the oldest versions could be
* marked for removal.
*
* The third mode could be the actual deletion of any artifacts which were marked for removal.
* The delay is to allow human intervention before wholesale deletion.
*
* The versionsToUse is an array of strings that are the start of builds that should be processed.
*
* There are two additional options. The first is the dryRun option. This way you can get
* an overview of what will be processed. If specified, no artifacts will be altered.
*
* Usage example
* groovy ArtifactoryProcess.groovy --dry-run --function mark --value production --must-have releasable --web-server http://yourWebServer/artifactory/ --domain <com.YourOrg> --repository libs-release-prod 1.0.1 1.0.2
*
* #author Brian Carr (snippets from Jettro Coenradie, David Carr and others)
*/
class ArtifactoryProcess {
public static final Set< String > validFunctions = [ 'mark', 'delete', 'clear', 'download', 'config', 'repoPrint' ];
public static final Set< String > validParameters = [ 'function', 'value', 'mustHave', 'targetDir', 'maxInState', 'webServer', 'repoName', 'domain', 'userName', 'password' ];
#Option(name='--dry-run', usage='Don\'t change anything; just list what would be done')
boolean dryRun;
#Option(name='--full-log', usage='Log miscellaneous steps for processing artifacts')
boolean fullLog;
// eg --function mark
#Option(name='--function', metaVar='function', usage="function to perform on artifacts")
String function;
// eg --value production
#Option(name='--value', metaVar='value', usage="value to use with function above, often required")
String value;
// eg --must-have releasable
#Option(name='--must-have', metaVar='mustHave', usage="property required before applying delete, mark, download or clear, optional")
String mustHave;
// eg --targetDir d:/temp/bin
#Option(name='--targetDir', metaVar='targetDir', usage="target directory for downloaded artifacts")
String targetDir;
// eg --maxInState MaxInState.csv
#Option(name='--maxInState', metaVar='maxInState', usage="name of csv file with states and max counts, optional")
String maxInState;
// eg --web-server 'http://artifactory01/artifactory/'
#Option(name='--web-server', metaVar='webServer', usage='URL to use to access Artifactory server')
String webServer;
// eg --repository 'libs-release-prod'
#Option(name='--repository', metaVar='repoName', usage='Name of the repository to scan.')
String repoName;
// eg --domain 'org/apache'
#Option(name='--domain', metaVar='domain', usage='Name of the domain to scan.')
String domain;
// eg --userName cleaner
#Option(name='--userName', metaVar='userName', usage='userName to use to access Artifactory server')
String userName;
// eg --password SomePswd
#Option(name='--password', metaVar='password', usage='Password to use to access Artifactory server.')
String password;
#Argument
ArrayList<String> versionsToUse = new ArrayList<String>();
class PathAndDate{
String path;
Date dtCreated;
}
class StateRecord {
String state;
int cnt;
List< PathAndDate > pathAndDate;
}
#SuppressWarnings(["SystemExit", "CatchThrowable"])
static void main( String[] args ) {
try {
new ArtifactoryProcess().doMain( args );
} catch (Throwable throwable) {
// Java returns exit code 0 if it terminates because of an uncaught Throwable.
// That's bad if we have a process like Bamboo depending on errors being non-zero.
// Thus, we catch all Throwables and explicitly set the exit code.
println( "Unexpected error: ${throwable}" )
System.exit(1)
}
System.exit(0);
}
List< StateRecord > stateSet = [];
Artifactory srvr;
RepositoryHandle repo;
private int numProcessed = 0;
String firstFunction;
String lastConfig;
void doMain( String[] args ) {
CmdLineParser parser = new CmdLineParser( this );
try {
parser.parseArgument(args);
if( function == 'config' && value == null ) {
throw new CmdLineException("You must provide a config.csv file as the value if you specify the config function.");
}
firstFunction = function; // Flag in case we recurse into config files, where did we start
if( function == 'config' ) {
processConfig();
return;
} else {
checkParms();
}
} catch(CmdLineException ex) {
System.err.println(ex.getMessage());
System.err.println();
System.err.println("groovy ArtifactoryProcess.groovy [--dry-run] [--full-log] --function <func> --value <val> --web-server http://YourWebServer --repository libs-release-prod --domain <com/YourOrg> Version1 ...");
parser.printUsage(System.err);
System.err.println();
System.err.println(" Example: groovy ArtifactoryProcess.groovy"+parser.printExample(ExampleMode.ALL)+" 1.0.1 1.0.2");
System.err.println();
System.err.println(" Supported functions include ${validFunctions}" );
System.err.println();
System.err.println(" Columns in config csv files can be ${validParameters}" );
return;
}
String stateLims;
if( maxInState != null && maxInState.size() > 0 ) stateLims = "(using stateLims)" else stateLims = "(no stateLims)"
println( "Started processing of $function with ${(value==null)?mustHave:value} $stateLims on $webServer in $repoName/$domain with $versionsToUse." );
withClient { newClient ->
srvr = newClient;
if( function == 'repoPrint' ) printRepositories();
else {
processRepo();
}
}
}
def processRepo() {
numProcessed = 0; // Reset count from last repo.
repo = srvr.repository( repoName );
processArtifactsRecursive( domain );
if( dryRun ) {
println "$numProcessed folders would have been $function[ed] with $value.";
} else {
println "$numProcessed folders were $function[ed] with $value.";
}
}
def processConfig() {
File configCSV = new File( value );
lastConfig = value; // Record which csv file we have last dived into.
Artifactory mySrvr = srvr; // Each line of config could have a different web server, preserve connection in case recursing
configCSV.withReader {
CsvIterator csvIt = CsvParser.parseCsv( it );
for( csvRec in csvIt ) {
if (fullLog) println("Step is ${csvRec}");
Map cols = csvRec.properties.columns;
String func = csvRec.function;
def hasFunc = cols.containsKey( 'function' );
def has = cols.containsKey( 'targetDir' );
if( cols.containsKey( 'function' ) && !noValue( csvRec.function ) ) function = csvRec.function ;
if( cols.containsKey( 'value' ) && !noValue( csvRec.value ) ) value = csvRec.value ;
if( cols.containsKey( 'targetDir' ) && !noValue( csvRec.targetDir ) ) targetDir = csvRec.targetDir ;
if( cols.containsKey( 'maxInState' ) && !noValue( csvRec.maxInState ) ) maxInState = csvRec.maxInState;
if( cols.containsKey( 'webServer' ) && !noValue( csvRec.webServer ) ) webServer = csvRec.webServer ;
if( cols.containsKey( 'repoName' ) && !noValue( csvRec.repoName ) ) repoName = csvRec.repoName ;
if( cols.containsKey( 'domain' ) && !noValue( csvRec.domain ) ) repoName = csvRec.domain ;
if( cols.containsKey( 'userName' ) && !noValue( csvRec.userName ) ) userName = csvRec.userName ;
if( cols.containsKey( 'password' ) && !noValue( csvRec.password ) ) password = csvRec.password ;
if( cols.containsKey( 'mustHave' ) ) mustHave = csvRec.mustHave; // Can clear out mustHave value
checkParms();
withClient { newClient ->
srvr = newClient;
processRepo();
}
srvr = mySrvr; // Restore previous web server connection
}
}
}
def checkParms() {
if( !noValue( maxInState ) ) {
stateSet.clear(); // Throw away any previous states from last step
File stateFile = new File( maxInState );
def RC = stateFile.withReader {
CsvIterator csvFile = CsvParser.parseCsv( it );
for( csvRec in csvFile ) {
String state = csvRec.properties.values[ 0 ];
String strCnt = csvRec.properties.values[ 1 ];
if( fullLog ) println( "State ${state} allowed ${strCnt}" );
int count = 0;
if( strCnt.integer ) count = strCnt.toInteger();
if( count < 0 ) count = 0;
// Iterator lies and claims there is a next when there isn't. Force break on empty state.
stateSet.add( new StateRecord( state: state, cnt: count, pathAndDate: [] ) );
}
}
}
String prefix;
if( firstFunction == 'config' && function != 'config' ) {
prefix = "While processing ${lastConfig} encountered, ";
} else prefix = '';
if( !validFunctions.contains( function ) ) {
throw new CmdLineException( "${prefix}Unrecognized function ${function}, function is required and must be one of ${validFunctions}." );
}
if( function == 'mark' && noValue( value ) ) {
throw new CmdLineException( "${prefix}You must provide a value to mark with if you specify the mark function." );
}
if( function == 'clear' && noValue( value ) ) {
throw new CmdLineException( "${prefix}You must provide a value to clear with if you specify the clear function." );
}
if( function != 'repoPrint' && noValue( domain ) ) {
throw new CmdLineException( "${prefix}You must provide a domain to use with the ${function} function." );
}
if( function == 'download' ) {
if( noValue( targetDir ) ) targetDir = '.';
}
if( noValue( webServer ) || noValue( userName ) || noValue( password ) || noValue( repoName ) ) {
throw new CmdLineException( "${prefix}You must provide the webServer, userName, password and repository name values to use." );
}
if( versionsToUse.size() == 0 && stateSet.size() == 0 && function != 'repoPrint' ) {
throw new CmdLineException( "${prefix}You must provide maxInState or a list of artifacts / versions to act upon." );
}
}
Boolean noValue( var ) {
return var == null || var == '';
}
/**
* Print information about all the available repositories in the configured Artifactory
*/
def printRepositories() {
Repositories repos = srvr.repositories();
List repoList = repos.list( RepositoryTypeImpl.LOCAL );
for( it in repoList ) {
println "key :" + it.key
println "type : " + it.type
println "description : " + it.description
println "url : " + it.url
println ""
};
}
/**
* Recursively removes all folders containing builds that start with the configured paths.
*
* #param path String containing the folder to check and use the childs to recursively check as well.
* #return Number with the amount of folders that were processed.
*/
private int processArtifactsRecursive( String path ) {
ItemHandle item = repo.folder( path );
// def RC = item.isFolder(); This lies, always returns true even for a file, go figure!
// def RC = path.endsWith('.xml'); // item.info() fails for simple files, go figure!
if( !path.endsWith( '.xml' ) &&
!path.endsWith( '.jar' ) &&
item.isFolder() ) {
Folder fldr;
try{
fldr = item.info()
} catch( Exception e ) {
println( "Error accessing $webServer/$repoName/$path" );
throw( e );
};
for( kid in fldr.children ) {
boolean processed = false;
if( stateSet.size() > 0 ) {
if( isEndNode( kid.uri )) {
processed = groupFolders( path + kid.uri );
}
} else {
versionsToUse.find { version ->
if( kid.uri.startsWith( '/' + version ) ) {
numProcessed += processItem( path + kid.uri );
return true; // Once we find a match, no others are interesting, we are outta here
} else return false; // Just formalize the on to next iterator
}
}
if( !processed ) {
processArtifactsRecursive( path + kid.uri );
}
}
}
/* If we are counting number in each state, our lists should be all set now */
if( stateSet.size() > 0 ) {
processSet();
}
return numProcessed;
}
private boolean processedThis( String vrsn, kid ) {
if( kid.uri.startsWith('/' + vrsn )) {
numProcessed += processItem( vrsn + kid.uri );
return true; // Once we find a match, no others are interesting, we are outta here
} else return false; // Just formalize the on to next iterator
}
// True if nodeName is of form int.int.other, could be one line, but how would you debug it.
private boolean isEndNode( String nodeName ){
int firstDot = nodeName.indexOf( '.' );
if( firstDot <= 1 ) return false; // nodeName starts with '/' which is ignored
int secondDot = nodeName.indexOf( '.', firstDot + 1 );
if( secondDot <= 0 ) return false;
String firstInt = nodeName.substring( 1, firstDot ); // nodeName starts with '/' which is ignored
if( !firstInt.isInteger() ) return false;
String secondInt = nodeName.substring( firstDot + 1, secondDot );
if( secondInt.isInteger() ) return true;
return false;
}
private boolean groupFolders( String path ) {
Map<String, List<String>> props;
stateSet.find { rec ->
ItemHandle folder = repo.folder( path );
if( rec.state.size() > 0 ) {
props = folder.getProperties( rec.state );
}
if( rec.state.size() <= 0 || props.size() > 0 ) {
PathAndDate nodePathDate = new PathAndDate();
nodePathDate.path = path;
nodePathDate.dtCreated = folder.info().lastModified;
rec.pathAndDate.add( nodePathDate ); // process this one
return true; // No others are interest, break out of iterator
} else return false; // On to next iterator
}
return true; // We always process all nodes which are end nodes
}
private boolean processSet() {
for( set in stateSet ) {
int del = set.cnt;
if( set.pathAndDate.size() < del ) {
del = set.pathAndDate.size() }
else {
set.pathAndDate.sort() { a,b -> b.dtCreated <=> a.dtCreated }; // Sort newest first to preserve newest
}
while( del > 0 ) {
set.pathAndDate.remove( 0 );
del--;
}
while( set.pathAndDate.size() > 0 ) {
numProcessed += processItem( set.pathAndDate[ 0 ].path );
set.pathAndDate.remove( 0 );
}
}
return true;
}
private int processItem( String path ) {
int retVal = 0;
if( fullLog ) println "Processing folder: ${path}, ${function} with ${value}.";
def RC;
ItemHandle folder = repo.folder( path );
Map<String, List<String>> props;
boolean hasRqrd = true;
if( !noValue( mustHave ) ) {
props = folder.getProperties( mustHave );
if( props.size() > 0 ) hasRqrd = true; else hasRqrd = false; // I like this better than ternary operator, you?
}
if( !hasRqrd ) return retVal;
switch( function ) {
case 'delete':
if( !dryRun ) RC = repo.delete( path );
retVal++;
break;
case 'download':
if( folder.isFolder() ) {
Folder item = folder.info();
item.children.find() { kid ->
if( kid.uri.endsWith('.jar') ) {
DownloadableArtifact DA = repo.download( path + kid.uri );
InputStream dlJar = DA.doDownload(); // Open Source
FileWriter lclJar = new FileWriter( targetDir + kid.uri, false ); // Open Dest
for( id in dlJar ) { lclJar.write( id ); } // Copy contents
if( fullLog ) println( "Downloaded ${path + kid.uri} to ${targetDir + kid.uri}." );
retVal++;
return true;
}
}
}
break;
case 'mark':
props = folder.getProperties( value );
if( props.size() == 0 ) {
PropertiesHandler item = folder.properties();
PropertiesHandler PH = item.addProperty( value, 'true' );
if( !dryRun ) RC = PH.doSet();
retVal++;
}
break;
case 'clear':
props = folder.getProperties( value );
if( props.size() == 1 ) {
if( !dryRun ) RC = folder.deleteProperty( value ); // Null return is success, go figure!
retVal++;
}
break;
default:
println( "Unknown function $function with $value encountered on ${path}.")
}
if( retVal > 0 ) println "Completed $function on $path with ${(value==null)?mustHave:value}.";
return retVal;
}
private <T> T withClient( #ClosureParams( value = SimpleType, options = "org.jfrog.artifactory.client.Artifactory" ) Closure<T> closure ) {
def client = ArtifactoryClient.create( "${webServer}artifactory", userName, password )
try {
return closure( client )
} finally {
client.close()
}
}
}