We recently switched from Ant to Buildr for building our projects. We use Ivy for dependency management, using the ivy4r Buildr extension. We have a local repository at the office which is used as a cache for public artifacts and in which we also publish our own artifacts.
Now for the problem: I'd like to be able to build my project when I do not have access to the office repository. Buildr has a flag to tell it to work offline (-o), but ivy4r does not seem to take this into account. Is there any way to make Ivy not try to download artifacts? I have them all available in the cache on my machine already.
Setting the cache timeout to eternal
You can set the cache property ${ivy.cache.ttl.default} to eternal this will set the TTL: so that the repository will not be checked for new revisions.
You could achieve this by calling ant with the following parameter:
ant -Divy.cache.ttl.default=eternal build
This is from the documentation:
Defines a TTL (Time To Live) rule for resolved revision caching. When
Ivy resolves a dynamic version constraint (like latest.integration or
a version range), it can store the result of the resolution (like
latest.integration=1.5.1) for a given time, called TTL. It means that
Ivy will reuse this dynamic revision resolution result without
accessing the repositories for the duration of the TTL, unless running
resolve in refresh mode.
...
The TTL duration can also be set to 'eternal', in which case once
resolved the revision is always use, except when resolving in refresh
mode.
Other references:
IVY-879 Implementation of this feature
Setting UseCacheOnly for the resolve task
The resolve task has the attribute useCacheOnly, which can be used to
force[s] the resolvers to only use their caches and not their actual
contents
Example:
<ivy:resolve file="path/to/ivy.xml" useCacheOnly="true/>
Related
I am looking for a solution to automatically increment a package version (not to be confused with an assembly version) after each build on CI server (particularly Atlassian Bamboo). Every appx package has a version defined in its manifest file (appxmanifest). Thus in order to increase the version a manifest must be edited before commit. I am considering different approaches to implement this. The first one makes changes in manifest and pushes it back to the repo.
Starts building a plan (in order to lock a build number)
Modifies manifest so that a revision is set to the current build number
Pushes changes to SCM (particularly Atlassian Stash). This step shouldn't trigger the next build.
Continues building the package (invoke MSBuild, UT and other tasks)
Cons
Leads to incorrect workflow on Bamboo: checkout -> push -> build
Each build makes a new commit
Another approach is to setup post receive Stash hook which would modify appxmanifest.
Cons Hard to keep a build number in sync with Bamboo.
Is there any other (cleaner and proper) way to achieve this?
ex-Stash developer here (not that it matters),
I would highly recommend not checking in derived/version information or files. It's going to cause you no end of problems (some of which you have pointed out in your question).
My advice - generate what information you need on the build. I don't know anything about appx packaging, but can you use a placeholder/property (like this) which can be resolved on the Bamboo build? For our builds we use the git hash and timestamp as the version, and in the past I've also used the job/build number (timestamp is better though).
As more food for thought - if that appx version is important for developers to see locally, and it becomes hard to match up with the Git version then you can also attach a Git tag/note to the commit in Bamboo as well. The nice thing about that is that anyone fetching from Git can easily see that extra metadata, but it doesn't result in extra commits for every build. If the appx version need to be based off the previous version then this makes it possible for the build scripts to inspect the previous commit and bump the version appropriately.
I hope that helps.
I would like to deploy snapshot builds from Bamboo to Artifactory. My repository's Handle Snapshots option is checked and it's Maven Snapshot Version Behavior is set to Unique. The repository's layout is gradle-default.
My goal is for a build plan to deploy an artifact at a location similar to the following:
repo-local:com.company/project/1.0-SNAPSHOT/project-1.0-20120612.101600.txt
In Bamboo I have a Artifactory Generic Deploy Task, configured with the following for the Edit Published Artifacts field:
project-1.0-SNAPSHOT.txt=>com.company/project/1.0-SNAPSHOT
However Artifactory rejects my build artifacts, saying The repository 'repo-local' rejected the artifact 'repo-local:com.company/project/1.0-SNAPSHOT/project-1.0-SNAPSHOT.txt' due to its snapshot/release handling policy.
How do I get Artifactory to accept the artifact and automatically replace SNAPSHOT with a timestamp in the filename?
Your problem is most likely the fact that the path you deploy to is not considered a valid integration revision by the layout you've selected (gradle-default).
The gradle-default layout expects integration revisions like:
org/module/1.0-12345678912345/module-1.0-12345678912345.jar
That is, for a 14 digit long time stamp to be appended after the base revision;
While your path contains SNAPSHOT instead of a 14 digit long timestamp.
If you want to have pattern like:
com.company/project/1.0-SNAPSHOT/project-1.0-20120612.101600.txt
You will have to customize the layout to accept -SNAPSHOT as the folder integration revision and modify your artifact to contain a timestamp as the file integration revision.
I'm guessing your assumption was that Artifactory will convert the non-unique integration revision to a unique one; Artifactory performs this conversion only when the repository is set to the default Maven layout and when the artifacts adhere to Maven's layout.
This is due to the fact the while Maven actually has defined standards for integration revisions, Gradle do not have such a standard; So basically, a Gradle revision could be practically anything.
On top of that, the concept of unique and non-unique integration revisions doesn't really exist in the Gradle world, it doesn't actually have any built-on functionality to support these features; and so when you see a Mavenized path in Gradle, it's just basically mimicking the pattern.
(This question is asked on Maven User mailing list too)
I have recently faced a strange problem, that I cannot even able to judge the cause or source of problem. It will be great if someone can give me some direction:
(The story may be a bit long)
I am using Nexus 1.8.0 as our company's repository manager. I use it as proxy of external repo, and hosting our own repository.
There are many repositories in Nexus. I have one repository group (let's call it PUBLIC) which groups all public repositories, including maven central, codehaus etc.
There is another repository group (let's call it EXT) which we put 3rd party artifacts.
In our project, we used org.codehaus.mojo:native2ascii-maven-plugin.
Due to a bug at that time, instead of using the publicly available org.codehaus.mojo:native2ascii-maven-plugin:1.0-alpha-1, I have fixed the bug and deploy it to our EXT repository, and called it org.codehaus.mojo:native2ascii-maven-plugin:1.0-alpha-1.1 (i.e. used a new version number 1.0-alpha-1.1 instead of 1.0-alpha-1)
This have been running fine for several years.
However recently a new developer tries to get the code and build, using Maven 2.2.1. Strange things happened: the build failed. By inspecting result of mvn -X clean install, it states that POM of native2ascii-maven-plugin:1.0-alpha-1.1 cannot be downloaded from PUBLIC, therefore it will use a default emtpy POM, which cause the build problem.
By inspecting the local repository, I found that only the JAR of native2ascii-maven-plugin:1.0-alpha-1.1 was downloaded. I am sure that there is no native2ascii-maven-plugin:1.0-alpha-1.1 in PUBLIC repository, and the SHA of the JAR matches with native2ascii-maven-plugin:1.0-alpha-1.1 in EXT. It seems that, Maven is capable to download the JAR correctly from EXT repo, but when it tries to download the POM afterwards, Maven mistakenly think that it should be downloaded from PUBLIC. Because PUBLIC do not contains 1.0-alpha-1.1, Maven assume there is no POM.
I have EXT repo defined before PUBLIC in my settings.xml. What even more strange is, I tried to block accessing in Nexus for native2ascii-maven-plugin from PUBLIC. Maven, instead of getting the POM from repository EXT, it get from central directly. At last I add PUBLIC as mirror for central, and Maven can build correctly, because EXT is the the only repo that contains native2ascii-maven-plugin. Maven seems tries to download the POM from every repository else which contains native2ascii-maven-plugin in despite of the version number, except from EXT
I simply cannot understand why this will happen. This have been used for years, and it used to be fine even several weeks before (I have other new developers, who can correctly download the plugin, several weeks ago). May anyone guide me the possible cause of the problem? I have neither changed anything in my repo, nor changed version of Maven. Why Maven's "download" behavior suddenly changed?
It's hard to say.
First my theory on why it no longer works. I am guessing this "worked for years" because at one time it worked, and afterwards everything was in your local repository (<home>/.m2/repository). Later, something broke, but you never noticed because you had everything local. The new developer did not have a populated local repository so when they built for the first time, they had failures.
Now my suggestion which may not work out for you. When using Nexus, I think its best to create a single "group" repository that links in all other repositories, and configure the group to order the priority of the linked repositories. So for you, in the group, you would list EXT first, then PUBLIC. Your POMs and/or settings would reference only the group repository. This may just duplicate what you are already doing through other means, but at least it is moving the ordering rules up into Nexus. I would rename your local repository (so you can revert back if necessary) and try re-building to see if everything resolves correctly.
You might want to consider a continuous build tool like Hudson that periodically deletes its own local repository so you can catch issues like this sooner.
At last I managed to find out the "cause" of the problem. It is due to my fault, combined with still-unknown behavior of Maven. I add this as an answer to ease future reference for other people.
They key problem is that I missed plugin version for this specific project (I did put corresponding pluginManagement for other projects, and other plugins for this project... I wonder how come I made this mistake this time)
The way to reproduce the problem:
A separate repository to store the plugin (in my case, org.codehaus.mojo:native2ascii-maven-plugin:1.0-alpha-1.1)
In project POM, add plugin, without version. For example,
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>native2ascii-maven-plugin</artifactId>
<plugin>
</plugins>
in settings.xml, avoid defining mirrors (i.e. the settings.xml contains list of repositories and pluginRepositories only)
With such setup, first purge the local repository. Then build the project. After build, inspect the directory in local repository for that plugin (in my case .m2\repository\org\codehaus\mojo\native2ascii-maven-plugin\1.0-alpha-1.1), you will find only the JAR presents, without corresponding POM. (Caused by Maven successfully get the plugin JAR corresponding to the pluginRepositories in settings.xml, but trying to get the POM from a weird location)
With the same setup, put the version in project POM, clean up the local repo, and build again. Everything is fine now.
The reason for work fine even for a recently clean CI environment, is probably due to other "correct" project made the plugin downloaded correctly, which can be used by this "incorrect" project. A periodic purge in local repository in CI won't necessary help much on this too because for that many projects, the chance is always very high for other "correct" project build earlier than that "incorrect" project.
The reason behind such behavior of Maven is still unknown, but at least in a "correct" POM (with plugin version correctly declared), Maven works fine. I will raise this as a issue for Maven though.
I'd start off by agreeing with SingleShot in that Continuous Integration - even a simple smoke test where you simply compile and run unit tests on the trunk - would have prevented you getting into the situation of assuming that the because the build works on one machine, it does not work on the other.
This have been running fine for several years.
That's the kicker with Maven repositories - all you need to do is download it once succesfully, and you'll be forever good to go. Just because it's been working successfully from your local repository doesn't mean it was working.
It is fine several months ago (coz I have migrated our CI server and I have a clean env to build, and everything is fine).
Interesting. So my theory would be then to go and make sure the new developer is set up correctly - that the settings.xml file is in place and is being read (I've had instances that the settings.xml is THERE, but in the wrong place!). It's a simple one, but Maven does not fail if there's no settings.xml, it just uses a default that may have you seeing ghosts.
You mentioned that you use maven 2.2.1 and I can only ask you to doublecheck, we had some strange behavior concerning downloading jars from internal repo that was caused by OSX Lion update that comes with maven3. Our fix was to redeploy affected project.
I'm learning how to augment my build with Ivy using a "brute force" method of just trying to get a few sample projects up and running. I've poured over the official docs and read several online tutorials, but am choking on a few terms that seem to be used vaguely, ambiguously and/or in conflicting ways somehow. I'm just looking for an experienced Ivy connoisseur to help bring some clarity to these terms for me.
"Resolution" Cache vs. "Repository" Cache vs. "Ivy" Cache
The "Ivy Repository", as opposed to my normal SCM which is a server running SVN
What's the difference between these 3 types of cache? What's the difference between the "Ivy Repository" and my SVN?
Thanks to anyone who can help!
"Resolution" Cache vs. "Repository" Cache vs. "Ivy" Cache
The ivy cache is basically a folder, where ivy stores artifacts and configurations. If not configured differently it can be found in UserHome/.ivy2
The ivy cache consists of the resolution cache and a repository cache.
The repository cache contains the artifacts from a repository, that were downloaded by ivy. It is caching the repository, so that ivy won't need to query the repository every time it tries to resolve/download an artefact. If it finds an suitable artifact in the repository cache it will not query the repository. Thus saving the cost to query the repository. If and how the cache is used is a bit more complicated and depends on the dependencies/configuration.
The resolution cache is a collection of ivy-specific files, that tell ivy how an artifact was resolved (downloaded).
The "Ivy Repository", as opposed to my normal SCM which is a server running SVN
A Repository in the ivy world is a location, which contains artifacts(jar) files. This can be the local filesystem or a web server. It has no versioning system. Each version of an artifact is contained in a seperate folder. You can't commit artifacts, you just add them to the file system. See the terminology
org\artifact\version1\artifact.jar
org\artifact\version2\artifact.jar
A repository is accessed via a resolver, which has to know the layout of the repository.
From the doc on caches:
Cache types
An Ivy cache is composed of two different parts:
the repository cache
The repository cache is where Ivy stores data downloaded from module repositories, along with some meta information concerning these artifacts, like their original location.
This part of the cache can be shared if you use a well suited lock strategy.
the resolution cache
This part of the cache is used to store resolution data, which is used by Ivy to reuse the results of a resolve process.
This part of the cache is overwritten each time a new resolve is performed, and should never be used by multiple processes at the same time.
While there is always only one resolution cache, you can define multiple repository caches, each resolver being able to use a separate cache.
Some of the artifacts in my local Nexus repository don't have the correct checksum. For example (wrong checksum):
cat central/org/codehaus/plexus/plexus-compiler-api/1.8/plexus-compiler-api-1.8.pom.sha1
95f3332c2bbace129da501424f297e47dd0e976b
vs (correct checksum):
sha1sum central/org/codehaus/plexus/plexus-compiler-api/1.8/plexus-compiler-api-1.8.pom
4c2947f7e2d09b6e13da34292d897c564f1f9828
It looks like I have a few artifacts in my repository that were downloaded when this bug was active.
Maven Central has the correct checksum (4c29...) now, but the checksums in my local Nexus repository remain stale. I don't know how to get my local repository to verify and / or re-download the correct checksum from central.
What is the correct way of fixing my local repository. There aren't too many artifacts with this problem, so I think I could (by hand) verify they still exist in central and delete them from my local repository. They should get re-cached with the correct checksums. Is there a better way?
Update:
I've looked at this more and I'm almost positive I know what the source of my problem is. One of the artifacts I'm having trouble with is this one (plexus-compiler-api:1.8):
In my repository, both the .pom and .pom.sha1 are timestamped as 29-Mar-2010. At central, the .pom is timestamped as 29-Mar-2010 while the .pom.sha1 is timestamped as 21-Apr-2010. I was reading about Nexus maintenance. I assume that, on 21-Apr-2010, Maven Central rebuilt metadata and verified checksums which fixed the incorrect .sha1 for the plexus-compiler-api:1.8 artifact.
According to the Sonatype link above, I should be able to expire the caches for Maven Central and have my local installation pull new copies of anything with newer timestamps than the originally cached artifacts. However, based on the behavior I've observed, I think it's only checking timestamps for artifact files, not checksum files.
As far as my local Nexus repository is concerned, I have the most recent version of the artifact (29-Mar-2010), so there's no need to re-download anything.
I've noticed my version of Nexus is quite old (1.5 vs 1.9.1), so I'll try updating and see if the newer version does a better job of expiring caches. If not, I'll probably see what the Sonatype guys think (maybe it's a bug?).
Nope, what you face is the defined behaviour of Nexus and Maven.
First, expiring caches does not delete anything from local cache of Nexus, it just marks them "old". The effect of marking items as "old" is shown on next incoming request asking for those same artifacts (if never asked for, the "old" artifacts just sits there). Meaning, expire cache alone will not cause Nexus to download remotely changed (newer) files. Nexus never downloads on it's own (if we leave out the index from this discussion). You have to force a client (Maven) ask for them – and that will result following chain of action: "cache content old", remote change detection and finally re-download and caching of the new file.
Next, what happens here is that Maven, since artifact (the JAR file) is not changed, not even asks for checksum file either, hence nothing "triggers" the "old" marked checksum refetch on Nexus side. Other to note, if we talk about released artifact (and Maven Central does contains released artifacts only), Maven will never re-check them, unless they are not present in local repository (once brought into local repository, Maven will never try to refetch them). Meaning, you need to remove them from local repository to be sure that Maven will ask for them from Nexus, and finally, that Nexus will detect the checksum file changes on remote and do what you actually want.
Re-download should happen, for example if you nuke your Maven's local repository and rebuild with a clean/empty one. In this case, Maven should ask for both, JAR artifact and checksum file – but from your description it's not clear how did you (or did you?) invoke Maven after expiring caches on Nexus.
Try this:
a) run expire caches on Nexus "Maven Central" proxy repository
b) nuke local repository (or just redirect it to a new clean folder by tampering ~/.m2/settings.xml
c) make Maven build your project, and it should refetch both, the JAR and checksum files (by using empty/nuked local repository)
Hope this explains some of the stuff you wrote.
Reference to JIRA issue discussing same thing.
This was a bug.
As explained by Tamas, when a proxied repository cache is expired, Nexus will check the remote repository for newer timestamps. The locally cached artifacts are essentially flagged dirty and the check for updated artifacts happens on demand as artifacts are requested from the local Nexus server.
Nexus (1.9.1) is making the assumption that if an artifact timestamp is unchanged, the checksums should be unchanged as well. Most of the time this will be true, but, due to the old bug in Maven that was deploying artifacts with incorrect checksums, there are rare cases where an artifact can be unchanged yet have an updated checksum.
I think the best way to deal with this for now will be to move any bad checksums and let Nexus try to re-resolve them the next time they are requested:
mv plexus-compiler-api.pom.sha1 plexus-compiler-api-1.8.pom.sha1.bak
Thanks for the help Tamas.