Creating temporary database that works across maven test phases? - sql

I've joined a project that has a lot of files with SQL statements for creating a database that is used for integration testing.
I'm wondering how I can use these files to create a database for unit testing
(using java and maven).
I can create a HSQL in-memory database for each unit test, or even use the spring jdbc embedded-database feature, but there's so many SQL statements to execute in the test setup that this is not scalable.
So I'd like to create a temporary database (that loads the SQL statements) at the start of the maven test phase, have the unit tests access this temporary database and perform various operations, then delete the temporary database at the end of the maven test phase.
I've looked at sql-maven-plugin which would allow me to do the test phase executions, but I'm not sure how to configure a temporary database that will be available across all unit tests. There's no server to connect to, and in-memory database will not work across multiple unit tests (I assume).
One option could be to use a unique temporary file, e.g. specifying the JDBC driver URL as jdbc:hsqldb:file:/path/to/temporary/file, but I'm not sure how to generate a unique temporary file in maven.
Any suggestions on how to do this, or if there's a better approach to take?
Update: I decide to use a file-based database created in target/db directory. I use the maven clean plugin to remove the target/db directory before tests are run, and the maven sql plugin to create the database from scripts.

For this case I have created the derby-maven-plugin. It's available from Maven Central, so you don't need to add any extra repositories or anything.
You could use it like this:
<project ...>
<build>
<plugins>
<plugin>
<groupId>org.carlspring.maven</groupId>
<artifactId>derby-maven-plugin</artifactId>
<version>1.8</version>
<configuration>
<basedir>${project.build.directory}/derby</basedir>
<port>1527</port>
</configuration>
<executions>
<execution>
<id>start-derby</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop-derby</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
For more info you can also check the USAGE.

Why not create an on-disk H2 database, and have each test access it? So long as the tests don't run in parallel or interact with each other, you don't need a server.
Even more so: just create memory databases in #Before and delete them in #After. Are you sure that's too slow?
In pre-integration-test, you could launch an H2 (or derby) server, and shut it down in post-integration-test.
You can write a maven plugin that uses session state to keep track of an embedded database service, but that's much the same as (3).

Related

Run static analysis tools in compile or package step in Maven2?

I'm researching using Maven for my team, and one of my goals for the build is to have static analysis tools run for every build, and perhaps fail the build for certain severe errors. However, in maven it seems that these are treated as reports and run on the site lifecycle. This presents a couple of problems as we don't intend to use the site for most projects, and it requires an extra step (or at least additional phase) to get the static analysis.
Is there a way to run static analysis tools in maven-2 in the compile or package phase (or equivalent)?
This can be done by configuring the plugin (findbugs for example) in the <build> section and configuring the <phase> in the <executions>:
<executions>
<execution>
<id>findbugs-run</id>
<phase>process-classes</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>

How to include a maven launch config in a maven archetype?

I got a liferay-portlet-project with a sample application/portlet that I want to become an archetype. Inside the project there is a folder containing two *.launch files to redeploy the webapp. Both have the following line which I have trouble with:
<stringAttribute key="org.eclipse.jdt.launching.WORKING_DIRECTORY" value="${workspace_loc:/rawportlet}"/>
where "rawportlet" is the project's name. If I change it manually to ${artifactId} this variable is not resolved when using the archetype to create a project. Resolving this variable during project-generation would be nice.
Is there a way to achieve this? Or a workaround? Thanks in advance for your help.
Workaround: write a maven goal that the user can run after using the archetype. So the steps would be (for example):
generate project from archetype
mvn archetype:generate -DarchetypeCatalog=local
do some post-generation cleanup (execute in project's base dir)
mvn antrun:run
So my code for this is in "pom.xml" in the archetype:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>default-cli</id>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<replace token= "rawportlet" value="${artifactId}" dir="runConfigs">
<include name="**/*.launch"/>
</replace>
</tasks>
</configuration>
</execution>
</executions>
</plugin>
The "runConfigs" directory is where the *.launch files are stored.
Credits to:
Full search and replace of strings in source files when copying resources
Maven, configure specific goal
I have this same problem, and I used a different solution that works okay (but isn't perfect either).
Use value="${workspace_loc}/${artifactId}" in your launch config.
This will work as long as people do an archetype:gen at the workspace root. This works better for me than the selected answer because running that post processing requires another launch configuration (which somewhat defeats the whole purpose).

Simple Mavenization of existing Ant build files

If you have an existing ant file, what is the best way to convert the project to Maven. I've checked out things like fAnt, but if I'm going to mess with this stuff, I might as well go full-bore for Maven. I expected something to exist that can just start the pom.xml for me based on the existing build.xml, but I haven't found anything yet. Suggestions?
I don't know any good automated way to do such a migration because things may just be too different so I would do it manually, step by step, and keep the existing ant build in parallel of the future new one until the whole migration is done (from both technical and human points of view).
First, refactor the existing Ant build to align it on Maven conventions:
Make things modular: if your existing build is a big monolithic build producing several artifacts from a single source tree, break it down into separate modules, one for each artifact.
Update directory structure: Maven comes with a standard directory layout and, while it is possible to customize this layout (i.e. to configure plugins for another layout), this is not really recommended and is more a source of troubles than benefits. So I'd move existing app sources, configuration files, tests, etc to match Maven's layout (e.g. src/main/java for application sources, etc).
Then, start to create the Maven build:
Create POMs for each module: Create a POM, declare external libraries as Maven dependencies (maybe add them to a corporate repository, using an enterprise repository is a good practice in an enterprise context anyway), add dependencies between modules.
Finalize the multi-modules build: Add parent(s) POM(s) and inheritance/aggregating relationships. Test that there is no regression with the created artifacts.
You could do this work in a separate VCS branch if you don't want to change anything until the work is done and create scripts to move things. And when ready, merge the Maven specific stuff and apply the scripts.
You could run the Ant script from Maven with the maven-antrun-plugin. Your pom.xml would look something like this:
<project>
...
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<dependencies>
<dependency>
<groupId>org.apache.ant</groupId>
<artifactId>ant-nodeps</artifactId>
<version>${ant-nodeps.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>init</id>
<phase>compile</phase>
<configuration>
<tasks>
<!-- Ant code goes here -->
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
</build>
</project>
That way you can start to move your dependencies into Maven, and reference them in the Ant script like so
${com.foo.bar:my-lib:jar}
Then just start slowly moving pieces of your Ant into pure Maven stuff.

Combined site/deploy goal for Maven

When running a Maven build on the CI server, I generate the site to publish the documentation and reports, and also deploy the artifact to the snapshot repository for use by other projects. To do this I run the following goals:
mvn clean site deploy
This means the unit tests are run twice, once for the site lifecycle and once for the deploy lifecycle. If I configure the site goal to be bound to the standard lifecycle the tests are still run twice, running the site goal always causes the tests to be run because of the #requiresDependencyResolution test annotation. This is fine if you're only creating the site, but in the context of a deploy it greatly increases the build time for no benefit.
I have a workaround that involves copying the SiteMojo (and the required parents) to a new plugin and removing the #requiresDependencyResolution test annotation from the copy.
This modified mojo will generate the reports without forcing the tests to be run again but I'd prefer a solution that doesn't involve any hacking of the site plugin. Is there a way to suppress the requiresDependencyResolution annotation?
I'm surprised this works - the #requiresDependencyResolution test tag doesn't actually trigger the tests being built - that should be one of the reports that you've included. Normally, I recommend running the site and the build in separate Maven executions in CI so you can get fast feedback on your build and publish the latest site only when that succeeds.
Another alternative is to run it as mvn clean deploy site, and choose the report-only mojo for surefire-report-maven-plugin (this is usually the report that is running the tests again). This will use the previous test results. Of course, another alternative is disabling that report altogether, since you likely get those results from another source such as your CI server anyway.
My current approach is to create a new plugin with copies of the relevant types from the maven-site-plugin. These types are identical to the standard versions except for changing the type name, the goal name and the removal of the #requiresDependencyResolution test annotation.
The copied types are:
org.apache.maven.plugins.site.AbstractSiteMojo
org.apache.maven.plugins.site.AbstractSiteRenderingMojo
The parent mojos are required so Maven can process the javadoc-based annotations (this shouldn't be required for Maven 2.2.0+).
org.apache.maven.plugins.site.SiteMojo
org.apache.maven.plugins.site.SiteJarMojo
These two are copied as SiteOnlyMojo and SiteJarOnlyMojo respectively, SiteJarOnlyMojo is changed to inherit from SiteOnlyMojo . Otherwise the only changes are to change the goal namess and remove the annotation.
So SiteOnlyMojo has:
* #goal site
* #requiresDependencyResolution test
changed to:
* #goal site-only
and SiteJarOnlyMojo has:
* #goal jar
* #phase package
changed to:
* #goal jar-only
* #phase package
These types are declared in a maven-plugin project with artifactId maven-site-only-plugin with a dependency declared on the proper site plugin.
To use this I define a profile (I don't want the reports running on every execution, only when -Psite is declared on the command line) and bind it to the prepare-package phase (prior to 2.1.0, you'd have to bind it to the package phase instead).
<profile>
<id>site</id>
<build>
<plugins>
<plugin>
<artifactId>maven-site-only-plugin</artifactId>
<version>1.0.0</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>jar-only</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<build>
<pluginManagement>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.2</version>
<executions>
<execution>
<goals>
<goal>site</goal>
<goal>deploy</goal>
</goals>
<phase>deploy</phase>
</execution>
</executions>
</plugin>
...
</plugins>
</pluginManagement>
</build>
This will automatically execute the site action when issuing a 'mvn deploy' command, as well as ensuring the testing suite is only executed once.

Wiping out Maven local repository on build machine

On a CI build server, the local Maven repository fills up the file system repetitively (after a few days).
What strategy are others doing to trim the local repository in such a case?
-Max
The Maven dependency plugin has a purge-local-repository goal that allows you to delete the dependencies for a given project from the local repository, if this is run say once a day on each project the snapshots will not accumulate.
Alternatively there's a more scorched-earth approach you could take. As the problem is typically the timestamped snapshot artifacts, you could use the maven-antrun-plugin to delete all files that match the resource collection pattern.
For example (note this might need some tweaking as I've done it from memory):
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<configuration>
<tasks>
<delete>
<fileset dir="${settings.localRepository}">
<include name="**/*.jar"/>
<exclude name="**/*.pom"/>
<exclude name="**/*.war"/>
<exclude name="**/*.ear"/>
<exclude name="**/*.md5"/>
<exclude name="**/*.sha"/>
<!--any other extensions?...-->
<!--match the timestamp pattern-->
<containsregexp expression="[0-9]{8}.[0-9]{6}-[0-9]+"/>
</fileset>
</delete>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
If you're using hudson, you can set up a scheduled job to just delete the entire repository once a day or something like that. I've got a job called hudson-maven-repo-clean which has this configuration:
Build / Execute shell: rm -rf ~hudson/.m2/repository
Build Triggers / Build periodically: 0 0 * * *
In addition to purge-local-repository (which reads to me like a nuclear option, as it only offers an excludes configuration as opposed to an explicit includes), take a look at the Remove Project Artifact mojo. I'm looking to implement it now, as my exact use case is to clear out large WAR and EAR snapshots that are being built on my CI (and sometimes workstation) machines.
We use especially for this purpose the build-helper plugin. In our company parent pom is the remove-project-artifact goal embedded in the profile for our hudson builds. This way all old versions of this artifact are removed prior to installing the currently build version.
...
<profile>
<id>hudson</id>
<activation>
<property>
<name>BUILD_TAG</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.7</version>
<executions>
<execution>
<id>remove-old-artifacts</id>
<phase>package</phase>
<goals>
<goal>remove-project-artifact</goal>
</goals>
<configuration>
<removeAll>true</removeAll>
</configuration>
</execution>
</executions>
</plugin>
...
Using removeAll set to true will wipe out all other snapshots except the one your working on. This can be dangerous as it may mean snapshots for a branch will be wiped out as well.
For instance if you have a snapshot 1.0.0.18-SNAPSHOT representing HEAD and snapshot 1.0.1.17-SNAPSHOT representing a branch, running this plugin with 1.0.0.18-SNAPSHOT build will wipe the 1.0.1.17-SNAPSHOt folder.
To get around this scenario the removeAll should be set to false.
We have employed a slightly different (and devious) technique. All artifacts that build "large things" (EARs, WARs, TARs) have their deploy location overriden like so:
<properties>
<discard-me-in-bit-bucket>file://${basedir}/target/_DELETEME</discard-me-in-bit-bucket>
</properties>
<distributionManagement>
<repository>
<id>upload-InternalSite</id>
<name>SoftwareLibrary External</name>
<url>${discard-me-in-bit-bucket}</url>
<layout>legacy</layout>
<uniqueVersion>false</uniqueVersion>
</repository>
<snapshotRepository>
<id>upload-InternalSite</id>
<name>Repository Name</name>
<url>${discard-me-in-bit-bucket}</url>
<layout>legacy</layout>
<uniqueVersion>false</uniqueVersion>
</snapshotRepository>
</distributionManagement>
This strategy causes the deploy goal to put things in the target directory, which of course is destroyed by the next CLEAN operation. To get even more aggressive, we have a postbuild step that does this:
find -type d -name '*_DELETEME' -exec rm -rf '{}' ';' -prune || echo $?
We employ yet one more strategy, too. In Hudson/Jenkins we provide a settings file to place the .m2 repository in the workspace for the job. This allows us to delete the entire repository before or after the job. It also makes artifacts visible in the workspace which aids in debugging some problems.
How big is the file system? We have 10gb allocated to builds and zap snapshots older than 30 days every night. That seems to work
Are you doing builds every X hours or when code changes? Switching to code changes will reduce the number of artifacts without reducing coverage.
Are you installing all snapshots locally? You don't need to do this in all cases. In most cases, just those snapshots that are actively developed dependancies need to be installed locally.
Are you installing EAR/WAR files locally? You probably don't need them either.
How many workspaces are you keeping? We use hudson and keep only the last 5 builds.