We recently added a second build machine to our build environment and began experiencing very odd occasional build failures.
I have two separate Maven build machines, A and B, each running Maven 2.2.1 and communicating to a shared Nexus 1.5.0 repository manager. My problem is that builds on B will occasionally fail because it refuses to download a newer version of a common dependency 'acme-1.0.0-SNAPSHOT' previously built by A and uploaded to Nexus.
Looking inside the local repositories on both machines I noticed some oddities in the repository metadata.
Machine A's acme\1.0.0-SNAPSHOT\maven-metadata-nexus.xml:
<metadata>
<groupId>acme</groupId>
<artifactId>acme</artifactId>
<version>1.0.0-SNAPSHOT</version>
<versioning>
<snapshot>
<buildNumber>1</buildNumber>
</snapshot>
<lastUpdated>20100525173546</lastUpdated>
</versioning>
</metadata>
Machine B's acme\1.0.0-SNAPSHOT\maven-metadata-nexus.xml:
<metadata>
<groupId>acme</groupId>
<artifactId>acme</artifactId>
<version>1.0.0-SNAPSHOT</version>
<versioning>
<snapshot>
<buildNumber>2</buildNumber>
</snapshot>
<lastUpdated>20100519232317</lastUpdated>
</versioning>
</metadata>
In Nexus's acme/1.0.0-SNAPSHOT/maven-metadata.xml:
<metadata>
<groupId>acme</groupId>
<artifactId>acme</artifactId>
<version>1.0.0-SNAPSHOT</version>
<versioning />
</metadata>
If I'm interpreting the metadata files correctly (documentation online is scant), it appears machine B believes it has a newer version of the acme dependency (based on buildNumber) despite the fact that machine A last built it 6 days after machine B did (based on timestamp). Nexus also appears to be unaware of a universally correct buildNumber.
How could this situation possibly arise? What could I do to prevent my builds from failing due to inconsistent metadata? Have you experienced anything similar?
Important notes:
Both build machines have settings.xml files where the updatePolicy is "always".
Nexus does indeed have the newer version of acme that was built by A. B simply refuses to download it.
A and B are the only machines uploading to Nexus.
Both servers share the same system time.
All processes involved have write privileges to the metadata files so that they can be updated as necessary.
I was unable to find any open Maven or Nexus issues describing this behaviour.
Our CI server (Atlassian Bamboo) prevents builds of the same artifact from happening concurrently, so some race condition while uploading to Nexus is rather unlikely.
It looks like you posted the wrong maven-metadata from Nexus, this looks like the one in the acme folder rather than the acme/1.0-SNAPSHOT folder. (it would have the build number and timestamp in there).
Anyway, have you tried adding -U to the maven build commands? It's possible that you've stumbled upon some maven bug respecting the always setting, but I'm sure -U works.
It took me a while, but I tracked down the underlying issue to maven bug MNG-4142.
Here's what happened:
My acme-1.0-SNAPSHOT (build 1) was installed on A and uploaded to Nexus. The project was next built on B where the newly built acme-1.0-SNAPSHOT (build 2) was installed and uploaded to Nexus, overriding build 1.
Then, when a build happened on the A machine that had acme-1.0-SNAPSHOT as a dependency, MNG-4142 kicked in. The repository metadata contained "true" which prevented A from downloading the more recent build 2 of acme-1.0-SNAPSHOT, and so maven built my project against the older build 1 which caused build failures. This was still the case even when -U was used.
As I mentioned on the issue, I'm quite surprised at this behaviour and struggle to think of how other distributed build environments work in the presence of this bug. We currently have some cron jobs that frequently change the "localCopy" metadata to false in order to get what I believe should be the default, and correct, behaviour.
Related
when I am doing development I often need to change a dependency, but I'm not ready to deploy my changes. For example, I'm working on project Foo and I realize I need to add a method to the common library. Before deploying this change to our internal repository, I would like to install the changes to common library (mvn install) and recompile Foo to use the common library in the local repository (note that I'm using all SNAPSHOT versions).
However, after I mvn install my common library, when I recompile Foo it doesn't use the new common library--it keeps using the latest SNAPSHOT of common library in the internal repository. If I deploy the changed common library, Foo picks it up immediately.
How can I get maven to look first in the local repository?
UPDATE: when the file is installed into the local repository, it gets a name like foo-1.0.0-SNAPSHOT.jar, but when I deploy it, it gets a timestamp foo-1.0.0-20111104.191316-23.jar. I think this is why the remote artifact gets pulled each time. Any idea why mvn install is not working like mvn deploy? Does it have to do with the fact that I have a snapshot repository set up for deploy?
By default, Maven checks for new versions of SNAPSHOT artifacts once per day. When it does this check, it will download SNAPSHOTS from remote repos that are newer than what you have locally. Either your artifact timestamps are out of sync and you're doing something to override Maven's update policy (like calling it with -U or setting the udpatePolicy to "always"), or else the local repository you're installing the artifact to isn't the same one you're subsequently running Maven against. What you're describing isn't typical Maven behavior. For a better answer, give more details in your question.
One indicator you can look for: after you install your common artifact, when you next compile Foo, does Maven download the common artifact again? If so, then it really is getting it from the remote, and you need to check your update settings. If not, then you have something strange going on locally.
You can try this option. This worked for me.
In your project's main pom.xml change 'snapshots' enabled setting to 'false'.
<repository>
<id>yourRepo</id>
<name>Repository</name>
<url>http://your.repo.com/repo</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
(This question is asked on Maven User mailing list too)
I have recently faced a strange problem, that I cannot even able to judge the cause or source of problem. It will be great if someone can give me some direction:
(The story may be a bit long)
I am using Nexus 1.8.0 as our company's repository manager. I use it as proxy of external repo, and hosting our own repository.
There are many repositories in Nexus. I have one repository group (let's call it PUBLIC) which groups all public repositories, including maven central, codehaus etc.
There is another repository group (let's call it EXT) which we put 3rd party artifacts.
In our project, we used org.codehaus.mojo:native2ascii-maven-plugin.
Due to a bug at that time, instead of using the publicly available org.codehaus.mojo:native2ascii-maven-plugin:1.0-alpha-1, I have fixed the bug and deploy it to our EXT repository, and called it org.codehaus.mojo:native2ascii-maven-plugin:1.0-alpha-1.1 (i.e. used a new version number 1.0-alpha-1.1 instead of 1.0-alpha-1)
This have been running fine for several years.
However recently a new developer tries to get the code and build, using Maven 2.2.1. Strange things happened: the build failed. By inspecting result of mvn -X clean install, it states that POM of native2ascii-maven-plugin:1.0-alpha-1.1 cannot be downloaded from PUBLIC, therefore it will use a default emtpy POM, which cause the build problem.
By inspecting the local repository, I found that only the JAR of native2ascii-maven-plugin:1.0-alpha-1.1 was downloaded. I am sure that there is no native2ascii-maven-plugin:1.0-alpha-1.1 in PUBLIC repository, and the SHA of the JAR matches with native2ascii-maven-plugin:1.0-alpha-1.1 in EXT. It seems that, Maven is capable to download the JAR correctly from EXT repo, but when it tries to download the POM afterwards, Maven mistakenly think that it should be downloaded from PUBLIC. Because PUBLIC do not contains 1.0-alpha-1.1, Maven assume there is no POM.
I have EXT repo defined before PUBLIC in my settings.xml. What even more strange is, I tried to block accessing in Nexus for native2ascii-maven-plugin from PUBLIC. Maven, instead of getting the POM from repository EXT, it get from central directly. At last I add PUBLIC as mirror for central, and Maven can build correctly, because EXT is the the only repo that contains native2ascii-maven-plugin. Maven seems tries to download the POM from every repository else which contains native2ascii-maven-plugin in despite of the version number, except from EXT
I simply cannot understand why this will happen. This have been used for years, and it used to be fine even several weeks before (I have other new developers, who can correctly download the plugin, several weeks ago). May anyone guide me the possible cause of the problem? I have neither changed anything in my repo, nor changed version of Maven. Why Maven's "download" behavior suddenly changed?
It's hard to say.
First my theory on why it no longer works. I am guessing this "worked for years" because at one time it worked, and afterwards everything was in your local repository (<home>/.m2/repository). Later, something broke, but you never noticed because you had everything local. The new developer did not have a populated local repository so when they built for the first time, they had failures.
Now my suggestion which may not work out for you. When using Nexus, I think its best to create a single "group" repository that links in all other repositories, and configure the group to order the priority of the linked repositories. So for you, in the group, you would list EXT first, then PUBLIC. Your POMs and/or settings would reference only the group repository. This may just duplicate what you are already doing through other means, but at least it is moving the ordering rules up into Nexus. I would rename your local repository (so you can revert back if necessary) and try re-building to see if everything resolves correctly.
You might want to consider a continuous build tool like Hudson that periodically deletes its own local repository so you can catch issues like this sooner.
At last I managed to find out the "cause" of the problem. It is due to my fault, combined with still-unknown behavior of Maven. I add this as an answer to ease future reference for other people.
They key problem is that I missed plugin version for this specific project (I did put corresponding pluginManagement for other projects, and other plugins for this project... I wonder how come I made this mistake this time)
The way to reproduce the problem:
A separate repository to store the plugin (in my case, org.codehaus.mojo:native2ascii-maven-plugin:1.0-alpha-1.1)
In project POM, add plugin, without version. For example,
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>native2ascii-maven-plugin</artifactId>
<plugin>
</plugins>
in settings.xml, avoid defining mirrors (i.e. the settings.xml contains list of repositories and pluginRepositories only)
With such setup, first purge the local repository. Then build the project. After build, inspect the directory in local repository for that plugin (in my case .m2\repository\org\codehaus\mojo\native2ascii-maven-plugin\1.0-alpha-1.1), you will find only the JAR presents, without corresponding POM. (Caused by Maven successfully get the plugin JAR corresponding to the pluginRepositories in settings.xml, but trying to get the POM from a weird location)
With the same setup, put the version in project POM, clean up the local repo, and build again. Everything is fine now.
The reason for work fine even for a recently clean CI environment, is probably due to other "correct" project made the plugin downloaded correctly, which can be used by this "incorrect" project. A periodic purge in local repository in CI won't necessary help much on this too because for that many projects, the chance is always very high for other "correct" project build earlier than that "incorrect" project.
The reason behind such behavior of Maven is still unknown, but at least in a "correct" POM (with plugin version correctly declared), Maven works fine. I will raise this as a issue for Maven though.
I'd start off by agreeing with SingleShot in that Continuous Integration - even a simple smoke test where you simply compile and run unit tests on the trunk - would have prevented you getting into the situation of assuming that the because the build works on one machine, it does not work on the other.
This have been running fine for several years.
That's the kicker with Maven repositories - all you need to do is download it once succesfully, and you'll be forever good to go. Just because it's been working successfully from your local repository doesn't mean it was working.
It is fine several months ago (coz I have migrated our CI server and I have a clean env to build, and everything is fine).
Interesting. So my theory would be then to go and make sure the new developer is set up correctly - that the settings.xml file is in place and is being read (I've had instances that the settings.xml is THERE, but in the wrong place!). It's a simple one, but Maven does not fail if there's no settings.xml, it just uses a default that may have you seeing ghosts.
You mentioned that you use maven 2.2.1 and I can only ask you to doublecheck, we had some strange behavior concerning downloading jars from internal repo that was caused by OSX Lion update that comes with maven3. Our fix was to redeploy affected project.
Some of the artifacts in my local Nexus repository don't have the correct checksum. For example (wrong checksum):
cat central/org/codehaus/plexus/plexus-compiler-api/1.8/plexus-compiler-api-1.8.pom.sha1
95f3332c2bbace129da501424f297e47dd0e976b
vs (correct checksum):
sha1sum central/org/codehaus/plexus/plexus-compiler-api/1.8/plexus-compiler-api-1.8.pom
4c2947f7e2d09b6e13da34292d897c564f1f9828
It looks like I have a few artifacts in my repository that were downloaded when this bug was active.
Maven Central has the correct checksum (4c29...) now, but the checksums in my local Nexus repository remain stale. I don't know how to get my local repository to verify and / or re-download the correct checksum from central.
What is the correct way of fixing my local repository. There aren't too many artifacts with this problem, so I think I could (by hand) verify they still exist in central and delete them from my local repository. They should get re-cached with the correct checksums. Is there a better way?
Update:
I've looked at this more and I'm almost positive I know what the source of my problem is. One of the artifacts I'm having trouble with is this one (plexus-compiler-api:1.8):
In my repository, both the .pom and .pom.sha1 are timestamped as 29-Mar-2010. At central, the .pom is timestamped as 29-Mar-2010 while the .pom.sha1 is timestamped as 21-Apr-2010. I was reading about Nexus maintenance. I assume that, on 21-Apr-2010, Maven Central rebuilt metadata and verified checksums which fixed the incorrect .sha1 for the plexus-compiler-api:1.8 artifact.
According to the Sonatype link above, I should be able to expire the caches for Maven Central and have my local installation pull new copies of anything with newer timestamps than the originally cached artifacts. However, based on the behavior I've observed, I think it's only checking timestamps for artifact files, not checksum files.
As far as my local Nexus repository is concerned, I have the most recent version of the artifact (29-Mar-2010), so there's no need to re-download anything.
I've noticed my version of Nexus is quite old (1.5 vs 1.9.1), so I'll try updating and see if the newer version does a better job of expiring caches. If not, I'll probably see what the Sonatype guys think (maybe it's a bug?).
Nope, what you face is the defined behaviour of Nexus and Maven.
First, expiring caches does not delete anything from local cache of Nexus, it just marks them "old". The effect of marking items as "old" is shown on next incoming request asking for those same artifacts (if never asked for, the "old" artifacts just sits there). Meaning, expire cache alone will not cause Nexus to download remotely changed (newer) files. Nexus never downloads on it's own (if we leave out the index from this discussion). You have to force a client (Maven) ask for them – and that will result following chain of action: "cache content old", remote change detection and finally re-download and caching of the new file.
Next, what happens here is that Maven, since artifact (the JAR file) is not changed, not even asks for checksum file either, hence nothing "triggers" the "old" marked checksum refetch on Nexus side. Other to note, if we talk about released artifact (and Maven Central does contains released artifacts only), Maven will never re-check them, unless they are not present in local repository (once brought into local repository, Maven will never try to refetch them). Meaning, you need to remove them from local repository to be sure that Maven will ask for them from Nexus, and finally, that Nexus will detect the checksum file changes on remote and do what you actually want.
Re-download should happen, for example if you nuke your Maven's local repository and rebuild with a clean/empty one. In this case, Maven should ask for both, JAR artifact and checksum file – but from your description it's not clear how did you (or did you?) invoke Maven after expiring caches on Nexus.
Try this:
a) run expire caches on Nexus "Maven Central" proxy repository
b) nuke local repository (or just redirect it to a new clean folder by tampering ~/.m2/settings.xml
c) make Maven build your project, and it should refetch both, the JAR and checksum files (by using empty/nuked local repository)
Hope this explains some of the stuff you wrote.
Reference to JIRA issue discussing same thing.
This was a bug.
As explained by Tamas, when a proxied repository cache is expired, Nexus will check the remote repository for newer timestamps. The locally cached artifacts are essentially flagged dirty and the check for updated artifacts happens on demand as artifacts are requested from the local Nexus server.
Nexus (1.9.1) is making the assumption that if an artifact timestamp is unchanged, the checksums should be unchanged as well. Most of the time this will be true, but, due to the old bug in Maven that was deploying artifacts with incorrect checksums, there are rare cases where an artifact can be unchanged yet have an updated checksum.
I think the best way to deal with this for now will be to move any bad checksums and let Nexus try to re-resolve them the next time they are requested:
mv plexus-compiler-api.pom.sha1 plexus-compiler-api-1.8.pom.sha1.bak
Thanks for the help Tamas.
I'd like to have a way in which 'mvn install' puts files in a repository folder under my source (checkout) root, while using 3rd party dependencies from ~/.m2/repository.
So after 'mvn install', the layout is:
/work/project/
repository
com/example/foo-1.0.jar
com/example/bar-1.0.jar
foo
src/main/java
bar
src/main/java
~/.m2/repository
log4j/log4j/1.2/log4j-1.2.jar
(In particular, /work/project/repository does not contain log4j)
In essense, I'm looking for a way of creating a composite repository that references other repositories
My intention is to be able to have multiple checkouts of the same source and work on each without overwriting each other in the local repository with 'install'. Multiple checkouts can be because of working on different branches in cvs/svn but in my case it is due to cloning of the master branch in git (in git, each clone is like a branch). I don't like the alternatives which are to use a special version/classifier per checkout or to reinstall (rebuild) everything each time I switch.
Maven can search multiple repositories (local, remote, "fake" remote) to resolve dependencies but there is only ONE local repository where artifacts get installed during install. It would be a real nightmare to install artifacts into specific locations and to maintain this list without breaking anything, that would just not work, you don't want to do this.
But, TBH, I don't get the point. So, why do you want to do this? There might be alternative and much simpler solutions, like installing your artifacts in the local repository and then copying them under your project root. Why wouldn't this work? I'd really like to know the final intention though.
UPDATE: Having read the update of the initial question, the only solution I can think of (given that you don't want to use different versions/tags) would be to use two local repositories and to switch between them (very error prone though).
To do so, either use different user accounts (as the local repository is user specific by default).
Or update your ~/.m2/settings.xml each time you want to switch:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
http://maven.apache.org/xsd/settings-1.0.0.xsd">
<localRepository>${user.home}/.m2/repository</localRepository>
<!--localRepository>${user.home}/.m2/repository2</localRepository-->
...
</settings>
Or have another settings.xml and point on it using the --settings option:
mvn install --settings /path/to/alternate/settings.xml
Or specify the alternate location on the command line using the -Dmaven.repo.local option:
mvn -Dmaven.repo.local=/path/to/repo
These solutions are all error prone as I said and none of them is very satisfying. Even if you might have very good reasons to work on several branches in parallel, your use case (not rebuilding everything) is not very common. Here, using distinct user accounts migh be the less worse solution IMO.
This is INDEED possible with the command line, and in fact is quite useful. For example, if you want to create an additional repo under your Eclipse project, you just do:
mvn install:install-file -DlocalRepositoryPath=repo \
-DcreateChecksum=true -Dpackaging=jar \
-Dfile=%2 -DgroupId=%3 -DartifactId=%4 -Dversion=%5
It's the "localRepositoryPath" parameter that will direct your install to any local repo you want.
I have this in a batch file that I run from my project root, and it installs the file into a "repo" directory within my project (hence the % parameters). So why would you want to do this? Well, let's you say you are professional services consultant, and you regularly go into customer locations where you are forced to use their security hardened laptops. You copy your self-contained project to their laptop from a USB stick, and presto, you can do your maven build no problem.
Generally, if you are using YOUR laptop, then it makes sense to have a single local repo that has everything in it. But to you who got cocky and said things like "why would you want to do that", I have some news...the world is a bigger place with more options than you might realize. If you are using laptops that are NOT yours, and you need to build your project on that laptop, get the resulting artifact, and then remove your project directory (and the local repo you just used), this is the way to go.
As to why you would want to have 2 local repos, the default .m2/repository is where the companies standard stuff goes, and the local "in project" repo is where YOUR stuff goes.
This is not possible with the command line client but you can create more complex repository layouts with a Maven repository server like Nexus.
The reason why it's not possible is that Maven allows to nest projects and most of them will reference each other, so installing each artifact in a different repository would lead to lots of searches on your local hard disk (or to failed builds when you start a build in a sub-project).
FYI: symlinks work in Windows7 and above so this kind of thing is easy to achieve if all your code goes in the same place in the local repo, i.e /com/myco/.
type mklink for details
I can see that you do not want to use special versions or classifiers but that is one of the best solutions to solve this problem. I work on the same project but different versions and each mvn install takes half an hour to build. The best option is to change the pom version appended with the change name, for example 1.0.0-SNAPSHOT-change1 that I'm working on thereby having multiple versions of the same project but with different code base.
It has made my life very easy in the long run. It helps run multiple builds at the same time without issues. Even during SCM push, we can skip the pom file from staging so there can always be 2 versions for you to work on.
In case you have a huge project with multiple sub-modules and want to change all the versions together, you can use the below command to do just that
mvn versions:set -DnewVersion=1.0.0-SNAPSHOT-change1 -DprocessAllModules
And once done, you can revert using
mvn versions:revert
I know this might be not what you are looking for, but it might help someone who wants to do this.
I'm trying to deploy the snapshot version of a 3rd party library to our local repo (for legacy reasons this is and old version which is no longer hosted at any online repo, and for the time being I can't replace it, hence I have to host it locally).
Now, I think I've misunderstood the SNAPSHOT-concept, so I'd be very thankful if somebody could set me straight. The jar I have is named foo-0.5.0-20090612.124.jar, and I try uploading it using maven deploy:deploy file to org.bar.foo under version 0.5.0.
The version dependency in my POM is 0.5.0-SNAPSHOT
Each time I upload the jar file to our local repo the number following the date in the filename is iterated (i.e to 125), but when I run maven install, maven attempts to download a jar with the previous number (i.e. 124).
So, is there any way of getting maven to download the correct snapshot version, or should I have no business uploading 3rd party SNAPSHOT files to our repo?
The use case you have is fine. I believe the best practice recommended by the Maven folks is that once you are uploading a SNAPSHOT version of a jar to a shared repository, you should stop treating it as a SNAPSHOT, and instead as a release.
This makes sense because you want people to depend on that specific version of that artifact. The artifact is not under active development and is not expected to change arbitrarily at any point in time and for that reason it should be considered a release. Just one to which you give your own personal fancy name, and one that someone else considered a SNAPSHOT.
When you ask for the dependency foo-0.5.0-SNAPSHOT, I don't believe Maven looks for foo-0.5.0-. I believe it literally looks for foo-0.5.0-SNAPSHOT in the repository, and that is why you are not finding it. (Not 100% sure about this though, as we don't timestamp our SNAPSHOTS. We simply continually overwrite -SNAPSHOT jars with newer versions as they are made.)
So, put foo-0.5.0-20090612.124.jar into your third party repository (which should only contain releases for the reason above). Then in your pom, reference the "released" artifact foo-0.5.0-20090612.124 explicitly, and everything should work fine.
I'm not sure I follow, you should have an artifact called myartifact.jar attached to a POM with a version of 0.5-SNAPSHOT.
When deploying, your repository should change the -SNAPSHOT into a timestamp and update the repository metadata to reflect that this is the most current SNAPSHOT version, so that it can properly serve a SNAPSHOT request.
Do you use a repository such as Archiva or Nexus or just a FTP/HTTP server?