I looking for ways to improve build time for my maven build.
I was following this suggestion (Speed up maven) to rename the target folder to a different name and then proceed with the install goal and thus avoid the time it takes to clean.
However this leaves me with a folder that still needs to be cleaned at some stage. I was hoping that I could automate the removal of this renamed folder as a second process that runs in parallel with the install phase.
I haven't been able to find any information on whether it is possible or not.
Cheers for the help,
Ferg
I'm not aware of an out-of-the-box feature that would do that. Also the antrun plugin seems not to have an option to fork the process and run in parallel.
maybe ant itself can do that within the build script?
Maven 3 supports parallel builds: https://cwiki.apache.org/MAVEN/parallel-builds-in-maven-3.html - that might help to speed things up (but depends on the plugins you are using).
Related
My Source code: 2 different branches from same repository. Both have 80% same code. Packages also same. But 20% classes are different in both branches. Its because of different features being developed in these branches.
My Intellij version is 2018.2.7 Community Edition
I cloned the repository in 2 different folders.
consider first one: C:\SourceCode\myRepo and move to branch1 from master
consider second one: C:\SourceCode\SecondCopy\myRepo and move to branch2 from master
Now I create 2 different projects in IntelliJ. One from C:\SourceCode\myRepo, Second from C:\SourceCode\SecondCopy\myRepo.
I build first project using git bash for windows using simple 'mvn clean install' command. (outside Intellij). Open Intellij. Run the application on Jetty. It goes good.
I have the second IntelliJ Project open as well in new Window. Now I build this second project in git bash using 'mvn clean install'(again outside Intellij). But as soon as I do that, the first project starts showing exceptions in code for the java classes which are not present in second project.
Even the application which was Already running using 'Run configuration of Intellij' in Jetty in First project, starts throwing class not found errors.
How can I resolve it and work on 2 projects having slightly different code but 80% same code, same packages etc. parallelly? What am I missing here? My intention is not to run Jetty parallelly. But to be able to compile these 2 projects independently and be able to Run Jetty in any one project anytime without interfering with other project.
These projects are big projects and hence, I want to initiate build and run for one project. Till the time its up and running, I want to be able to work on the other one parallelly in a way that, this work after compiling/auto building in this project doesn't impact the other one where I just started the application.
It is killing my productivity. Please help!
Use different Maven coordinates for the projects.
Someone told me that... Its happening because I am using maven and after building, maven keeps the project libraries that it builds in local repository. So both the projects after building are trying to keep the jars in the same folder from where Jetty is reading as well. Hence there is interference.
I have not tried. but If I keep 2 separate local repositories for both the projects, it should solve, but with this all the dependencies will also be unnecessarily duplicated. So the solution is as mentioned by #Andrey is that I keep the maven coordinates separate for both the projects, so that 2 separate jars are created for both. Thanks Andrey. I wont necessarily change maven coordinates for the all modules but for one where there is difference of code in both the projects. Hope this answer helps someone.
We are currently using bamboo to build and test our software. Now our build plans are just a bunch of task: execute this bat, execute that bat etc. Created with the Bamboo UI.
It happens that over months/years the build plan needs adjustments:
Parallelize jobs
Add extra jobs
Change some tasks
But this will break when we try to build an older version of the software. Some scripts (called from bamboo task) are not-existing in older versions.
At my previous employer we used Jenkins pipelines where the content of the build and test was just a file present in the sources repo.
Now with bamboo it appears you can use Bamboo Specs. From I read you create specs file and when you run this, it which will create build plan. But I don't see a relation to cater for changing build plans over time (changing steps).
For example the Bamboo Specs of develop are used to build all Plan Branches (e.g. Pull Requests). So if you want to change the build in a PullRequest, you first need to merge this into develop, the Bamboo Spec of develop updates the Build Plan. Not possible to test this before merging.
Question: How can you make scripted buildplans in Bamboo, where every branch of develop can a have possible other way of building?
We have it now setup as:
Buildplan 'Product A': plan branches: develop, release_x, release, y
Buildplan 'Product A PullRequest': plan branches: feature/*
Edit: supported in 7.0: https://confluence.atlassian.com/bamboo/enhanced-plan-branch-configuration-996709304.html
Old answer:
I found Atlassian documentation: https://jira.atlassian.com/browse/BAM-19620. They call it 'divergent plan branches'. No support, there is a feature request.
As of 15-4-2019:
Atlassian Update – [11 April 2019] Hi everyone,
Thank you for your votes and thoughts on this issue.
We fully understand that many of you are dependent on this
functionality.
After careful consideration, we've decided to prioritise [this
feature] on Bamboo roadmap. We hope to start development after our
current projects are completed.
Expect to hear an update on our progress within the next 6 months.
To learn more on how your suggestions are reviewed, see our updated
workflow for server feature suggestions.
Kind regards,
Bamboo Team
Question: How can you make scripted build plans in Bamboo?
To make scripted build plans in Bamboo, you have to use bamboo specs. Since you are already familiar with Jenkins, bamboo specs work exactly like Jenkinsfile by automating your pipeline. The benefit of using this is that it lives in your source code and the changes you make to this file in your source code automatically changes your plan(pipeline) when bamboo build is triggered.
This is how I script build plans in bamboo:
I add my bamboo.yml file under the root of my repo. But currently, I use git subtree and my bamboo specs live in there. But you don't have to do this. The below link provides you with a simple approach.
Link my repo to bamboo
Tell bamboo to scan for bamboo specs in the repo
Make commit and push
https://confluence.atlassian.com/bamboo/tutorial-bamboo-specs-yaml-stored-in-bitbucket-server-941616819.html
If I have to make changes to the plan in the future, I edit the bamboo specs file then commit and push.
I had the same problem and unfortunately had to go through an unpleasant choice
Backporting the build script
This is not necessarily feasible everywhere, but I managed to make it work somehow for my project.
The idea is: treat the build script as a C#/Java interface, or better as a contract.
As soon as your branches do not provide significant changes in building the software, e.g. your desktop app becomes a web app, or you switch from Ant to Gradle, you can handle this.
Assuming my application is always a web application to be released as a jar on JFrog Artifactory, I have identified the following steps that are common to all maintained versions:
Use javac to build the jar of all modules
Use gulp to build the Javascript resources
Run JUnit from the repository
Baptize 💒 the artifacts with a version number obtained with a tricky algorithm
Push the artifacts to JFrog Artifactory
So the idea is that I had taken my Ant build script and mostly rewrote it in order to do the same tasks on different versions of the application. I started doing the changes from an older version, not maintained anymore, as an excercise. In fact, my official Git branches look like release/x.y.z where semver is x.y.z.k and newer bugfix-builds are built from the head of any x.y.z release.
So I took release/3.10.0 branch and rewrote Ant. I am currently testing with a manually created Bamboo plan
Stage: Compile
ant clean ivy-retrieve compile jar #builds the jar in a job
ant gulp-install gulp-prod zip #creates javascript resources
Stage: Test
ant run-junit
Manual Stage: Release
ant baptize ivy-release #tags the artifact using ${bamboo.jira.version} and pushes to JFrog Artifactory
What I am going to do with Yaml
Since the build script is the same, but specific tasks (e.g. Java compiler version) may change in different versions, I can create a very single Yaml script that rules them versions all.
I will then merge release/3.10.0 => release/3.10.1 => release/3.10.2 ... release/3.11.2 by merging the conflicts
Personal experience
Tonight I am struggling in making the JUnit tests work as I also chose to backport my testing framework to the older version of the project. I accept that a few tests will fail because older and non-maintained versions contain bugs. For me this is a way to prove that the system works.
Indeed, diverging branches are a great idea, but I am forced to use Bamboo 6 in my office
I have 2 builds of the same project on the same Hudson/Jenkins server (they are running different build profiles). The both build when polling SCM, but I sometimes get the following error:
The plugin 'org.apache.maven.plugins:maven-checkstyle-plugin' does not exist or no valid version could be found
This only started once I added the second build, so I'm assuming there is some sort of conflict. Does anyone know what's going on here?
thanks,
Jeff
Are your builds running at the same time, with the same Maven instance and the same local Maven repository? If so, then maybe you run into a filelock on that plugin jar. I have the same problem here, if two jobs are accessing the same Maven artifact at the same time, they fail, because Maven can't handle that.
I solved that with two Maven instances, with a separate local repository.
It could also help, if you set the VCS polling of the second job a minute after the first job.
I am installing to Maven repository our internal files using mvn install:install command. All Jars installed in such way have version name added automatically as a suffix. Since we have many batch scripts with the Jars names it is very inconvenient for us. How this auto-renaming can be switched off?
You can't. Dependency resolution in Maven works because Maven has conventions and the naming of artifacts is one of them. So you can't turn off the way maven install artifacts (and you actually don't want to).
The common way to handle scripts (bat/sh) is to put them in your source tree (e.g. in src/main/bin) and to create a distribution of your project with the assembly plugin. When building your assembly, you can rename artifacts, filter distribution files, etc. That would be the right place to do such kind of things.
it's tempting to want to remove the version number, but I'd recommend keeping it on if at all possible.
Instead of removing the version numbers, maybe you could use Maven's resource filtering capability to manage your batch scripts? Maven can manage your batch scripts by replacing placeholders such as ${project.version} inside the bat script with the current version number from the pom.xml. That way you can ensure that the batch script is running the correct (expected) version of the code.
You could probably force maven to deploy a jar without a version, but if you do so, you lose some of the main benefits of having Maven manage your build in the first place, imo. For example, just yesterday I was asked to maintain a java project not managed by maven with a bunch of non-versioned jar files inside a lib directory. There's no way to know where any of the jars came from.
It might be a little bit of a headache up front, but if you're using maven, you might as well jump in 100% (again, from my experience).
I'm currently trying to reduce the time taken to compile and unit test projects in TeamCity.
Currently my project takes about between 5 to 8 minutes to build.
What it does is:
Clean any existing files
Compile each project
Create the installer
Once that is done the unit tests are kicked off and it takes about 2 minutes to run.
What it does is:
Clean any existing files
Compile each project
Run all the unit tests
Now, running the unit tests only takes 5 seconds....so the clean and compile step takes about 2 minutes....and the create installer step about 3 to 6 minutes.
My first question is: Is there any way to configure team city so that it doesn't have to do the clean and compile steps again when it runs the unit tests. I believe the main reason why we have it that way is because the project build and unit tests can be run by different build agents.
My second question is: Is 5 to 8 minutes a reasonable time for a project build to take? Are there ways to optimize the compile of the projects in the solution as well as the installer creation?
Please let me know if there are any additional details I can supply that might help you to point me in the right direction of either optimising the build or just leaving things as they are.
Update to answer some questions from Nate:
When you run the build outside TeamCity, does it do a clean/compile again before the unit tests? No, because when you run it on your own machine you can specify which parts of the build script to run. We have sections for clean, compile, unit-test etc which get specified differently in team city for project build vs unit test running.
If so, can you make the build not do that before the tests? The reason we don't do this at the moment is because when different build agents are used the files needed for the unit test would not be available and thus would fail.
What are you using to build your project? we use Nant to build our project, wix for the installer, and NUnit and NCover for the unit tests and coverage reports.
What source control system are you using? Subversion
Regarding #1: When you run the build outside TeamCity, does it do a clean/compile again before the unit tests? If so, can you make the build not do that before the tests?
What are you using to build your project? What source control system are you using?
Another option consider is having multiple build configurations. At my work, we have several configurations. One that runs after each commit. It does a clean/compile and some quick tests. We then have a nightly config that does a clean/compile and all the tests. Can you do something like that?
As a rule of thumb you should keep the build that is triggered on commit as short as posible, so you should try to refactor the build configuration into a slim on commit build, and run the current configuration once every hour or so.