I have a master Ivy project that others include in their project via a svn:externals property. The project contains the Ivy jar, the default ivysettings.xml file that connects to our project, and a few Ant macros that allows me to standardize the way we build jars, etc. (For example, users use <jar.macro> vs. <jar>. The <jar.macro> uses the same parameters, but also automatically embeds the pom.xml in the jar and adds in Jenkins build information into the Manifest).
We also use Jenkins as our continuous integration system. One of the things I want to do is to clean the Ivy cache for each build, so we don't have any jar issues due to cache problems. To do this, I've setup my ivysettings.xml file to define a separate cache for each Jenkins Executor:
<ivysettings>
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
I originally used the <ivy:settings> task to configure our projects with Ivy. However, all of the Jenkins executors were using the same Ivy cache which caused problems. I switched from <ivy:settings> to <ivy:configure> and the problem went away. Apparently, <ivy:configure> sets up Ivy immediately (and thus sets up the caches correctly) while <ivy:settings> doesn't set Ivy up until <ivy:resolve> is called.
I've seen some emails on Nabble about <ivy:configure> being deprecated (or maybe not). I see nothing in the Ivy online documentation stating <ivy:configure> is being deprecated.
So, when would you use <ivy:settings> vs. <ivy:configure>. In my case, since I needed separate caches for each Jenkins executor, I needed to use <ivy:configure>, but is there a reason I might use <ivy:settings> over <ivy:configure>? And, is <ivy:configure> deprecated?
here's what I found:
<ivy:settings> is newer and the preferred way.
<ivy:configure> may or may not be deprecated.
<ivy:settings> doesn't set my Ivy settings until <ivy:resolve> is called while <ivy:configure> sets all Ivy settings as soon as the task is executed.
The last one is my issue. Since I have parallel Jenkins builds going on, and I want to start out each build with a completely clean cache, I use customized cache settings depending upon the Jenkins executor number. The caches are labeled cache-0 through cache-5.
However, since <ivy:settings> isn't executed until I call <ivy:resolve>, my customized cache settings aren't picked up. I call <ivy:cleancache> before I call Ivy resolve which causes the builds to clean out a common cache. Hilarity ensues. Using <ivy:cofnfigure> fixes this problem.
Related
I can't get Ivy to update cache when snapshot dependencies are updated. The resolver (to has the following settings:
<url name="xxx" m2compatible="false"
checkmodified="true" changingMatcher="regexp"
changingPattern=".*-SNAPSHOT.*">
An example artifact filename (in Artifactory) is:
my-jar-1.999-SNAPSHOT.jar
A detailed Ant log of resolve includes:
[NOT REQUIRED] com.myorg#my-module;1.999-SNAPSHOT!my-jar.jar
There is no POM on the artifact.
The resolver is underneath a chain resolver; they both have all the relevant attributes set. I have read https://issues.apache.org/jira/browse/IVY-938 and https://issues.apache.org/jira/browse/IVY-1221, including all the comments, and AFAICT (perhaps incorrectly!) none of the workarounds are relevant.
Should I give up on snapshots and just use explicit versions with "integration.latest" dynamically versioned dependencies? I fear this may end up failing when we have integration builds happening for multiple major versions. At that point we'll need to split the major versions out into separate repositories, or put the major build number in the artifact name, or something equally clunky, just to make "integration.latest" work.
I'm not a fan of using the url resolver when talking to Maven repository managers.
The problem is Maven has special and rather unique handling for snapshot revisions..... The url resolver is better suited for use against ivy respositories.
I use Nexus, but the following should also apply to Artifactory. The following settings file sets up Maven Central and my two hosted repositories (Maven repositories come in two flavours, release or snapshot):
<ivysettings>
<settings defaultResolver="repos" />
<resolvers>
<chain name="repos">
<ibiblio name="central" m2compatible="true"/>
<ibiblio name="my-releases" m2compatible="true" root="https://myhost/releases"/>
<ibiblio name="my-snapshots" m2compatible="true" root="https://myhost/snapshots"/>
</chain>
</resolvers>
</ivysettings>
You'll notice I'm using the ibilio resolver which has internal logic to decipher Maven's special Snapshot handling.
When I require a snapshot revision I think declare it explicitly as follows:
<ivy-module version="2.0">
<info organisation="myOrg" module="Demo"/>
<dependencies>
<dependency org="myOrg" name="myModule" rev="2.7-SNAPSHOT"/>
..
</dependencies>
</ivy-module>
Under the hood the ibilio resolver is reading the Maven repository meta data files to determine which timestamped artifact should be fetched from the snapshot repository.
Update
I would suggest reading the following presentation:
Continuous delivery with Maven
It outlines pretty well the Maven philosophy separating releases from dev builds (or snapshots). It also explains one of the very clunky aspects of Maven... Two different ways to publish artifacts...
I suspect what you're trying to do is along the lines of the author which is setup a CD pipe-line. In that case every build is a potential release and should be treated as such (No dynamic dependencies which are allowed by snapshots).
I would suggest limiting snapshots to developer only builds. Only deploy release candidates. The problems with this approach will be in managing lots and lots of releases. Some of the repository managers (Nexus, Artifactory, Archiva) offer "staging" features which make it possible to certify or discard releases that don't pass your quality toll-gates.
Update 2
If you are using ivy to publish snapshots into a Maven repository then there are a couple of issues:
Ivy doesn't support the publication of snapshots with timestamps
Ivy doesn't update the Maven module's metadata.xml file
In my opinion time-stamped files is one of the killer features of using snapshots in the first place. With ivy it's only possible to provide the latest file (overwriting the previous latest file).
There are work-arounds to address these issues:
As suggested in the second link you can ignore metadata completely (setting the "useMavenMetadata" attribute to false) and default back to ivy's older mechanism of comparing file names. This only fixes the problem for ivy clients.
The repository manager should be able to regenerate the metadata files (Nexus at least has a task to do this).
Use the Maven ANT task.
The last suggestion is not as crazy as it seems. Firstly it's the only way I know to support timestamped snapshots and secondly the Maven client appears to do a lot of extra processing (updating the module metadata) that is largely undocumented.
After days of struggle...
The problem was that for
checkmodified="true" changingMatcher="regexp"
to work on a <resolver>, it has to be on every resolver in the hierarchy line - all parent <chain> resolvers and the <url>, <local>, or <ibiblio> resolver at the bottom.
I'm having problems understanding how latest.integration works.
I have an example that is not giving the output mentioned in:
http://ant.apache.org/ivy/history/latest-milestone/tutorial/defaultconf.html
which says, the local resolver has precedence over other resolvers, regardless of the time of publishing.
My ivysettings.xml goes like this:
<resolvers>
<chain name="download-chain" returnFirst="false" >
<url name="nexus-repo" m2compatible="true" >
<artifact pattern="${nexus}/[organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]" />
<ivy pattern="${nexus}/[organisation]/[module]/[revision]/[type]s/[artifact].[ext]" />
</url>
<resolver ref="local" />
</chain>
</resolvers>
here I declare that I have a nexus url repository and a reference to the default local. I use this chain when I want to resolve my dependencies.
I build the designated artifact and publish it to local with status "integration" with a revision "HEAD" (something like SNAPSHOT for us), first using the local resolver:
<ivy:publish
overwrite="yes"
update="true"
artifactspattern="${project.dist.dir}/[artifact]-[revision](-[classifier]).[ext]"
resolver="local"
settingsRef="ivy.nexus"
/>
and rebuild it again and publish it to nexus repository:
<ivy:publish
overwrite="yes"
update="true"
artifactspattern="${project.dist.dir}/[artifact]-[revision](-[classifier]).[ext]"
resolver="publish-url"
forcedeliver="true"
settingsRef="ivy.nexus"
/>
I have another project that declares the previous artifact as dependency with revision "latest.integration".
I expect that the artifact should be downloaded from the local repository regardless of the order of the declared resolvers. But that's not the case. The artifact downloaded is always of the resolver mentioned first. Changing the name of the "local" resolver didn't affect anything. The order is always what matters.
I tried appending changing="true" to my dependency. It didn't help.
In this question:
Ivy: Forcing local snapshot for dependency
The Asker mentions an even different behavior, namely that the very latest is picked up (the order of resolvers doesn't even matter).
SO to wrap it up and sorry for prolongation:
How to get an artifact:
1) always the latest.integration (the most recent) regardless of location.
2) always from local even if there's a more recent integration version in other locations.
3) Am I that clueless?
I would recommend reading the following answer on publishing artifacts to Nexus
how to publish 3rdparty artifacts with ivy and nexus
Use the ibiblio resolver, it's much simpler.
A second piece of advice is to have a clear separation between an integration and a release builds, within your ANT logic. The former can use a timestamp as it's revision, whereas the latter needs to have a strategy for maintaining an incrementing revision number (that's a whole different question). Maven calls these "SNAPSHOT" or "Release" builds and implements two different types of repository to support them.
A final piece of advice is to avoid using the local repository, unless, that is where you decide to store your integration builds. Ivy maintains a local cache of downloaded artifacts, it's rarely worth the effort or maintaining a local repo.
I could manage to make the order insignificant after all.
I'm not sure how far I should've gone but:
I used the latest="latest-time" in the chain resolver as well as in the URL resolver.
This wasn't however enough and when I debugged the code, I found that each resolver judged by its own 'latest'.
So I overrode the local repository like this:
<filesystem name="local" latest="latest-time" >
<ivy pattern="${ivy.local.default.root}/${ivy.local.default.ivy.pattern}"/>
<artifact pattern="${ivy.local.default.root}/${ivy.local.default.artifact.pattern}"/>
</filesystem>
Just discovered strange behavior of Ant Ivy cache and want to ensure that is not a bug but was implemented by intention.
I publish my module to the local repository, then ivy retrieves it to the cache while building another module. Good.
But if you navigate into cache directory (~/.ivy2/cache/[organisation]/[module]/) you'll see two flavors of module's ivy.xml file:
ivy-VERSION.xml.original
ivy-VERSION.xml
The first one is exactly the same as I have in my repository, while the second is obviously modified. For example it has status="release" (in the repository it is "integration"), publication timestamp is also changed (to the moment cache is updated).
Could somebody confirm this is correct behavior of the cache?
Very late to the party here, but wanted to capture this in case anyone else found this post.
I ran into a similar problem where the ivy.xml file was modified and the configuration information stripped way. Turns out the problem was that I had a period in the configuration description. Changing from "Java 1.7" to "Java 17" solved the problem and stopped the file from being modified.
<configurations>
<conf name="base" description="base dependencies that all configurations rely upon"/>
<conf name="apilegacy" description="dependencies and publication for the java 15 client" extends="base"/>
<conf name="api" description="dependencies and publication for the java 17 client" extends="base"/>
I've read the Ivy docs and a few other tutorials but am now trying to actually use it in a project for the first time and I am immediately presented with some roadblocks.
For the practice, I would like to write all my own config (XML) files. Not sure where to put ivy.xml, ivyconf.xml or ivy-settings.xml: do I put them in the same directory as my build.xml?
Besides ivy.xml, ivyconf.xml and ivy-settings.xml, are there any other config files that I should know about? Where do I place those?
Is the IvyDE just a graphical Eclipse plugin that graphically edits ivyconf.xml or does it edit other files?
Thanks for any input - it's been surprisingly difficult finding good info on this amazing tool!
You need only one file ivysettings.xml. You could place it beside build.xml or in any desired project directory. Optionally you could use one or more properties files for different uses.
Besides ivysettings.xml remeber about ivy cache directory. Often the best solution for all dependency resolution problems is to clean ivy cache by deleting this directory.
IvyDE allows you to resolve project dependencies inside Eclipse. IvyDE creates new classpath container named ivy.xml. Inside this container you'll find resolved libraries.
And I recomend you to use local or intranet ivy repository with Eclipse, because if resolving takes some time it will slow down loading of your projects.
For example I placed my ivysettings.xml that I use in my projects. As you can see for Eclipse there is eclipse-ivy.properties file. In this file I order Ivy to use local repository inside Eclipse. In Ant script I use remote repository by default, but there is an option to choose local repository.
<?xml version="1.0" encoding="UTF-8" ?>
<ivysettings>
<settings defaultResolver="${ivy.resolver}" />
<statuses default="development">
<status name="release" integration="false"/>
<status name="integration" integration="true"/>
<status name="development" integration="true"/>
</statuses>
<resolvers>
<ssh name="remote" checkconsistency="true" checkmodified="true" descriptor="required">
<ivy pattern="ssh://***/home/ivy/repository/[organisation]/[module]/[revision]/ivy.xml"/>
<artifact pattern="ssh://***/home/ivy/repository/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"/>
</ssh>
<filesystem name="local">
<ivy pattern="${user.home}/.local-ivy-repository/[organisation]/[module]/[revision]/ivy.xml"/>
<artifact pattern="${user.home}/.local-ivy-repository/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"/>
</filesystem>
</resolvers>
</ivysettings>
elipse-ivy.properties:
ivy.resolver=local
In general, it's best to create a single standardized ivy-settings.xml file and host it in a location where your builds can get to it easily. This file will usually be the same for all projects and rarely change once you've got it setup properly. In my development group, we host the ivy-settings.xml file on a web server so that it can easily be referenced by URL.
We build with Ant so we've also come up with a standardized ant build file which can either be hosted somewhere and imported into an Ant build script or simply copied to a project's directory if any kind of customization is necessary.
I've found that it's best to leave the Ivy configuration files as is and do any kind of customization by overriding the default values in your build scripts and/or ivy.xml files. Just about anything you'd need to change can be overridden. I recommend leaving the Ivy configuration files unchanged because it makes it much easier for other users to build your projects. If you've got a bunch of changes to your config files, any other developer wishing to build your project will have to replicate those changes on their local Ivy installs just to run a local build. That's bad mojo.
The IvyDE plugin provides editors for your ivy.xml and ivy-settings.xml files. According to the documentation, it also provides ways to automatically download your project's dependencies, includes a custom console and a reverse dependency explorer to help troubleshoot and resolve dependency conflicts. I only use the editors, though, so I can't vouch for the other tools.
Like Alexey, I'd recommend using a repository manager with Ivy. Artifactory does the job nicely; It's easy to setup and rock solid once it's running.
I agree that the documentation leaves much to be desired; it took me several weeks of experimentation to really grasp how all the pieces fit together. In the end, it was entirely worth it. Once you've got all the kinks ironed out, a well tuned artifact management system is truly something beautiful to behold.
I have a local Artifactory repository in which I have two jars for commons-logging: one for version 1.0.4 and one for version 1.1.1. I'm experimenting with using Ivy to download the older one with an ant task (with the proper dependency tag in ivy.xml), and then I change the "rev" attribute of this dependency tag to 1.1.1.
When using ivy:resolve in ant, this newer jar gets successfully downloaded to my cache, but the older one is not deleted automatically, and I'd like to make this happen.
I can't figure out how to do so after looking at the Ivy documentation; does anyone know how to get Ivy to delete old versions of artifacts when newer ones are downloaded, either with the resolve task or something else?
There's really no issue with having the "older" jar in your cache. If your project doesn't need the older jar, Ivy will simply ignore it. The old jar takes up about 50 kilobytes in your system. In the age of Terabyte drives, it's not worth the time and effort to free the space.
The Ivy cache is just that: A cache. It is for ALL of your projects that use Ivy. If an older project requires that 1.0.4 version of the commons-logging jar, it will already be in the Ivy cache and doesn't have to be downloaded, thus, like a good caches, it saves you time and effort.
You can do an <ivy:cleancache> but that really scrubs your entire cache: Everything will be deleted. It will get rid of the older 1.0.4 version of the jar, but it also will get rid of the 1.1.1 version of the jar too. Of course, Ivy will just download it the next time its requested, but that means waiting for Ivy to download all of your jars.
By the way, that's one big problem with <ivy:cleancache>: If you're doing multiple builds with Ivy and one cleans the cache on you, you'll end up breaking the other builds. Normally, this isn't an issue with individual users, but it can be for a Continuous Integration system.
If you look at the Ivy documentation, it does say that an <ivy:cleancache> is the same as:
<delete dir="${ivy.cache.dir}" />
You could use that as a basis for removing older jars by using a date selector:
<delete dir="${ivy.cache.dir}">
<date datetime="01/01/2010 12:00 AM" when="before"/>
<include name="*.jar"/>
</delete>
However, that uses the modified date of the jar and not the date the jar was created.
Ignore your Ivy cache directory and don't worry about it. If the Ivy cached does get too big, you can delete it and Ivy will simply redownload new jars as required.
I always include an ANT "realclean" target that additionally purges my ivy cache.
<target name="clean">
<delete dir="${build.dir}"/>
</target>
<target name="realclean" depends="clean">
<ivy:cleancache/>
</target>
Doco