Resolving Dependencies in creating JAR through SBT assembly - jvm

I want to create a big Jar file. for which am trying to use SBT ASSEMBLY. I installed sbt-assembly from GitHub and this answer. When I ran sbt assembly, I got this error:
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/UserName/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-2.5.0.v201103041518.jar:javax/servlet/SingleThreadModel.class
/home/UserName/.ivy2/cache/org.mortbay.jetty/servlet-api/jars/servlet-api-2.5-20081211.jar:javax/servlet/SingleThreadModel.class
To solve this, I followed User's README page and this is the code he suggests.
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("org", "apache", xs # _*) => MergeStrategy.last
case PathList("javax", "servlet", xs # _*) => MergeStrategy.last
case PathList("com", "esotericsoftware", xs # _*) => MergeStrategy.last
case PathList("project.clj") => MergeStrategy.last
case PathList("overview.html") => MergeStrategy.last
case x => old(x)
}
}
Even after adding this code, I get the same error mentioned before. Please let me know what I am missing. Any help on this error will be appreciated. Thanks!
Update 2 :
Added the exlusion rule as per the link given,
libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "0.8.0-incubating","com.codahale" % "jerkson_2.9.1" % "0.5.0","org.skife.com.typesafe.config" % "typesafe-config" % "0.3.0").map(_.exclude("javax", "servlet"))
Update 3:
I can locate the library that is causing the issue.
| +-org.apache.avro:avro-ipc:1.7.4
| | +-io.netty:netty:3.4.0.Final (evicted by: 3.5.4.Final)
| | +-io.netty:netty:3.5.4.Final
...
...
| | +-org.mortbay.jetty:jetty-util:6.1.26
| | +-org.mortbay.jetty:jetty:6.1.26
| | | +-org.mortbay.jetty:jetty-util:6.1.26
| | | +-org.mortbay.jetty:servlet-api:2.5-20081211
| | |
| | +-org.mortbay.jetty:servlet-api:2.5-20081211
| | +-org.slf4j:slf4j-api:1.7.2
...
...
| +-org.eclipse.jetty:jetty-server:7.6.8.v20121106
| | +-org.eclipse.jetty.orbit:javax.servlet:2.5.0.v201103041518
Update 4 : Fix
So adding MergeStrategy did solve the problem. Even though I had quite a few dependencies, more than 10, adding MergeStrategy for each one of them individually solved the issue.

I think you are trying to treat the symptoms, but your problem actually isn't assembly: Your project has a library twice in two different versions on the class path (javax.servlet).
If these versions are binary compatible (which I don't know), you would be fine by excluding one of the two occurrences in your build file like so. If they incompatible, you will need to unroll your dependency graph (a good way to do this might be the sbt-dependency-graph plugin) and try to find matching versions.
In any case, it might be useful to (at least transitorily) keep the libraries in the project folder. If you add retrieveManaged in ThisBuild := true to your sbt build file, all the libraries will be found in <project-root>/lib_managed. This allows you to see which jars are actually there.
EDIT: Showing the dependency graph:
Add to project/plugins.sbt:
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")
Add to build.sbt:
net.virtualvoid.sbt.graph.Plugin.graphSettings
Then run sbt dependency-graph.

You can also refer This Tech Blog
Creating a single JAR for Spark project using sbt-assembly
This post is about how to create a fat jar for spark streaming project using sbt plugin. sbt-assembly is a sbt plugin to create a fat JAR of sbt project with all of its dependencies.

Related

Intellij exclude single file from module output

How to exclude a single file from module output? I have seen the field "excluded files" in the module configuration (File -> Project structure -> modules) and tried to enter a pattern like that one:
path\to\file\*Server*
in order to exclude the class
path.to.file.MyServer
but it doesn't work
You should go to Settings | Build, Execution, Deployment | Compiler | Excludes and add your file name to the list.

Intellij Idea compile output including non-class file?

I had a project that source code including non-class file in package(eg:.sql,.ftl). I want to generate output including them but the artifacts always contains class file. How can I include them?
It's a web application, and below is source structure(java code part):
src
|
main
|
java
|
package1
|
Class1.java
Class2.java
sql1.sql
sql2.sql
template1.ftl
The output only contains compiled class.
I found a solution:
Project Structure->Module:at the bottom, Exclude files:*.java
Project Structure->Artifacts:Output Layout:add 'java' directory contents to classes

CMakeLists.txt cross-referencing each other

I am trying to migrate a project from a custom build script to cmake. The source structure looks roughly like this:
src
|
+-CMakeLists.txt
|
+-generated
| |
| +-CMakeLists.txt
| |
| +-database
| |
| +-...
|
+-main
|
+-CMakeLists.txt
|
+-database
|
+-...
The source files in generated/database get autogenerated. cmake seems to be capable of this (that's not part of the question), but I wonder how I can make it build stuff in the right order. main/database contains the framework that is used in the autogenerated files. However, there are other folders in main that depend on the generated sources. If I structure the top-level CMakeLists.txt like this:
add_subdirectory("generated")
add_subdirectory("main")
I cannot refer to main/database in generated/CMakeLists.txt as dependency.
Overall, I have the impression that cmake forces me to structure my files according to their dependencies, but I want to preserve the current layout - the dependencies in the project are far too complex to map them onto a file system hierarchy.
Should I just avoid add_subdirectory and write everything in the top-level CMakeLists.txt? It seems like this should be possible. Or is there another way to solve this?
Overall, I have the impression that cmake forces me to structure
my files according to their dependencies
If CMake has information that target A depends on target B (for example
target_link_libraries(A B)) B will be build first, then A.
If you use generated sources CMake can't get dependency information
and you need to provide it using add_dependencies.
From documentation:
Adding dependencies with this command can be used to make sure one target
is built before another target.

Automate AssemblyFileVersion Incrementation using GIT

Ok I understand that this is probably not conventional, but that aside: I am using the AssemblyFileVersion as kind of my "Build Name" string. It is formated like this:
' File Version information for an assembly consists of the following four values:
'
' Year
' Month
' Day
' Commit Number for that day
'
' Build Name can either be alpha | beta | hotfix | release
' alpha - is a development buildname with rapid changing API
' beta - is a production build for our beta users
' hotfix - is a production version with a bug fix
' release - is a standard issue production version.
<Assembly: AssemblyVersion("0.8.3")>
<Assembly: AssemblyFileVersion("13.10.24.3")>
<Assembly: AssemblyBuildName("alpha")>
Unfourtunately I am having to adjust the AssemblyInfo.vb EVERY TIME I do a git commit. Now I know GIT actually stores the commits in like a log file several places in the .git directory. My question is: is there anyway to automate this file to read from the git files to see what the year/month/day/commit#ForThatDay and automatically adjust the AssemblyFileVersion (or even a custom Assembly Attribute)?
I would use git describe in order to get an id representing the tag/SHA1 of the current commit, and integrate it in your Assembly file.
v1.0-2-g2414721-DEV
^ ^ ^ ^
| | | \-- if a dirtyMarker was given, it will appear here if the repository is in "dirty" state
| | \---------- the "g" prefixed commit id. The prefix is compatible with what git-describe would return - weird, but true.
| \------------- the number of commits away from the found tag. So "2414721" is 2 commits ahead of "v1.0", in this example.
\----------------- the "nearest" tag, to the mentioned commit.
It is similar to "Automatically versioning Android project from git describe with Android Studio/Gradle", but to be adapted to vb.net.
Or you can have "fake revision number".
For a more complete build Assembly file generation, see that maven plugin "maven-git-commit-id-plugin" (again, to be adapted to a vb.net build).
It can generate a file as complete as:
{
"branch" : "testing-maven-git-plugin",
"describe" : "v2.1.0-2-g2346463",
"commitTime" : "06.01.1970 # 16:16:26 CET",
"commitId" : "787e39f61f99110e74deed68ab9093088d64b969",
"commitIdAbbrev" : "787e39f",
"commitUserName" : "Konrad Malawski",
"commitUserEmail" : "konrad.malawski#java.pl",
"commitMessageFull" : "releasing my fun plugin :-)
+ fixed some typos
+ cleaned up directory structure
+ added license etc",
"commitMessageShort" : "releasing my fun plugin :-)",
"buildTime" : "06.01.1970 # 16:17:53 CET",
"buildUserName" : "Konrad Malawski",
"buildUserEmail" : "konrad.malawski#java.pl"
}
That illustrates how you can ask the git repo for all kind of different information (not just the date, but the branch, committer, commit message, ...).
See the DescribeCommand.java for more details on the implementation.

MSBuild 4.5 is ignoring project dependencies

I have a VS2010 solution file with over 20 projects in it, and some of the projects have dependencies on other projects from within the solution.
I also have multiple build configurations set up for different purposes, but I've trimmed down the projects to be built to just include the bare minimum number of projects.
For example, I have three libraries (A, B, and C), a website project, and a website deployment project (VS2010). The website has references to libraries A and B, and B in turn has a reference to C. My build configuration only has the website and the deployment project checked. When I check the project dependencies from the solution's properties, the website correctly lists the libraries, and B shows C, as expected.
When I run the build against my build config from within VS2010, it works completely fine, but when I run MSBuild on the solution specifying my configuration (as follows), it just results in a bunch of errors.
msbuild.exe mysolution.sln /t:Build /p:configuration=MyConfig
Here's an example of the errors I get:
Services\IService.cs(11,63): error CS0246: The type or namespace name 'Priority' could not be found (are you missing a using directive or an assembly reference?)
I noticed this happening on my build server (TeamCity v7.1.2), but I can reproduce it on multiple machines, and I narrowed it down to an issue with MSBuild, itself.
It only started happening after I installed .NET 4.5 (and 2 security patches), so I uninstalled it, reinstalled .NET 4.0 (with patches) since it was also removed, then tried the same command, and it worked just fine.
This leads me to believe that something was changed or broken in MSBuild with .NET 4.5, but nothing in their documentation seems to talk about this kind of change.
MSBuild 4.5 documentation: http://msdn.microsoft.com/en-us/library/hh162058.aspx
I've even tried passing BuildProjectDependencies=true to MSBuild, and it comes up stating that it skipped the other projects because they were not selected in the configuration manager, which is correct and intentional.
The only way I got it to work with MSBuild 4.5 was to go back and select the projects that were being skipped, but since the actual solution is a bit more complex with the dependency chain, I don't want to have to try to manage the configurations manually each time we update the solution with new projects or dependencies. It SHOULD be automatic.
Any ideas with what I'm doing?
If you still not resolved this issue, lets try some blind shots:
Msbuild have confirmed bug when it sometimes generates wrong build order based on followed dependencies. Try to build not entire solution but exact project you want to build - like msbuild.exe YourWebsiteProject.csproj /t:Clean;Build /p:configuration=MyConfig. Is the problem still persists ?
Ensure that YourWebsiteProject and your libs (B and C) have proper references - on project, not on dll in another project folder (simplest way to fix that - remove from B reference to C and re-add it again, just ensure that you are adding project reference and not browsing to bin\Debug for very C.dll). Is issue still there ?
If you could provide detailed or even diagnostic msbuild log (add to your msbuild command line following switches /ds /v:diag and then share teamcity full build log somewhere or pipe command line log to file) or some sample projects set where I can reproduce this behaviour - it could help a lot with issue resolving.
I thought I'd update my previous answer, as I've spent a lot of time and effort creating my own workaround to this problem. The work around is a bit more comprehensive than simply living with the problem, but I've attempted to both eliminate the issue and insulate ourselves against future shocks like this.
MSBuild has been demoted from working with solutions, configurations or otherwise. MSBuild is just asked to compile projects in isolation. The order this happens is calculated by a Powershell script that parses ours solutions and projects to work out the best Just-In-Time build execution plan.
Key to this (and I think you might find helpful) are the following snippets:
Identifying my solutions
I have a list of all the solutions in my platform, and I essentially iterate over each of these.
$buildPlan = (
#{
solutions = (
#{
name = "DataStorage"
namespace = "Platform.Databases"
},
#{
name = "CoreFramework"
},
#{
namespace = "Platform.Server"
name = "Application1"
},
#{
namespace = "Platform.Server"
name = "Application2"
},
#{
namespace = "Platform.Client"
name = "Application1"
}
)
})
I have some logic that helps to translate this into actual physical paths, but its very bespoke to our needs, so I won't list it here. Sufficed to say, from this list, I can find the .sln file I need to parse.
Parsing the solution file for projects
With each solution, I read the .sln file and attempt to extract all the projects contained within that I will need to build later.
So firstly, identify all projects in my
$solutionContent = Get-Content $solutionFile
$buildConfigurations += Get-Content $solutionFile | Select-String "{([a-fA-F0-9]{8}-([a-fA-F0-9]{4}-){3}[a-fA-F0-9]{12})}\.(Release.*)\|Any CPU\.Build" | % {
New-Object PSObject -Property #{
Name = $_.matches[0].groups[3].value.replace("Release ","");
Guid = $_.matches[0].groups[1].value
}
} | Sort-Object Name,Guid -unique
And then translate this into a nice list of projects that I can iterate over later.
$projectDefinitions = $solutionContent |
Select-String 'Project\(' |
ForEach-Object {
$projectParts = $_ -Split '[,=]' | ForEach-Object { $_.Trim('[ "{}]') };
$configs = ($buildConfigurations | where {$_.Guid -eq $projectParts[3]} | Select-Object Name)
foreach ($config in $configs)
{
$santisiedConfig = if ([string]::IsNullOrEmpty($config.Name)){"Release"}else{$config.Name}
if ($projectParts[1] -match "OurCompanyPrefix.")
{
New-Object PSObject -Property #{
Name = $projectParts[1];
File = $projectParts[2];
Guid = $projectParts[3];
Config = $santisiedConfig
}
}
}
}
Load the Visual Studio project
From my parsing of the solution file, I now have a list of projects per solution, which crucially contains the relative File Path from the solution root to find the project.
$projectDefinition = [xml](Get-Content $csProjectFileName)
$ns = #{ e = "http://schemas.microsoft.com/developer/msbuild/2003" }
$references = #();
1) Identifying external project references
$references += Select-Xml -Xml $projectDefinition -XPath "//e:Project/e:ItemGroup/e:Reference" -Namespace $ns | % {$_.Node} | where {$_.Include -match "OurCompanyPrefix" -and $_.HintPath -notmatch "packages"} | % {$_.Include}
2) Identifying internal project references
$references += Select-Xml -Xml $projectDefinition -XPath "//e:Project/e:ItemGroup/e:ProjectReference" -Namespace $ns | % { $_.Node.Name }
3) Following "Post-Build" events as external references
$references += Select-Xml -Xml $projectDefinition -XPath "//e:Project/e:PropertyGroup/e:PostBuildEvent" -Namespace $ns | where {(!([String]::IsNullOrEmpty($_.Node.InnerText)))} | % {
$postBuildEvents = $_.Node.InnerText.Split("`n")
$projectsReferencedInPostBuildEvents = $postBuildEvents | Select-String "\(SolutionDir\)((\w|\.)*)" | % {$_.Matches[0].Groups[1].Value}
if ($projectsReferencedInPostBuildEvents -ne $null)
{
Write-Output $projectsReferencedInPostBuildEvents | % { $matchedProject = $_; ($releaseConfiguation | ? {$_.File -match $matchedProject}).Name }
}
}
And, since we're at it, get some basic output information too
This is handy when it comes to iterating my list of projects to build, as know where to push the output, or where to find the output of a dependent.
$assemblyName = (Select-Xml -Xml $projectDefinition -XPath "//e:Project/e:PropertyGroup/e:AssemblyName" -Namespace $ns).Node.InnerText
$outputPath = (Select-Xml -Xml $projectDefinition -XPath "//e:Project/e:PropertyGroup[contains(#Condition,'Release|')]/e:OutputPath" -Namespace $ns).Node.InnerText
And at the end of it all
We just need to make sure we don't have any duplicates, so I record just the distinct dependencies of this particular code project:
$dependendents = #();
if ($references -ne $null)
{
$buildAction.project.dependencies += $references | where {(!([string]::IsNullOrEmpty($_))) -and ($_ -match "OurCompanyPrefix\.(.*)")} | % { $_.ToLower()} | Select -unique
}
I would hope this provides you with enough information for parsing your SLN and PROJ files. How you would choose to capture and store this information I think would depend entirely up to you.
I'm in the middle of writing quite an in-depth blog post about this, which will contain all the trimmings and framework I've eluded to above. The post isn't ready yet, but I will be linking to it from an earlier post : http://automagik.piximo.me/2013/02/just-in-time-compilation.html - Since this change by Microsoft nearly derailed this work!
Cheers.
In .NET 4.5, the default value of the OnlyReferenceAndBuildProjectsEnabledInSolutionConfiguration property in C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Common.targets was changed from false to true. As its name implies, this property causes MSBuild to ignore references to projects that are excluded from the build configuration.
(It's possible that Microsoft made this change in response to a Connect bug I filed against MSBuild 4.0, even though I warned them that the change breaks the build.)
The workaround is simple: Set the property back to false in the first <PropertyGroup> section of each of your projects:
<PropertyGroup>
...
<OnlyReferenceAndBuildProjectsEnabledInSolutionConfiguration>false</OnlyReferenceAndBuildProjectsEnabledInSolutionConfiguration>
</PropertyGroup>
I've encountered the exact same problem. The two work-around's I've managed to find are both unacceptable for the long term, but they do overcome the initial issue of getting old our build processes working with the new 4.5 stack.
Swap the project references with file references
Create compound build configurations
I've opted for #2, as file references mean the developers would lose out on real-time intellisense etc.
The compound configurations are simply this:
Release Server -> All Server Projects
Release Consumer -> "Release Server" + Client Projects
The problem seems to be, that if a project is not included in the current/active build configuration, that it will not include it as a referenced dependency. So, by adding the dependencies into the configuration, the projects will at least compile.
Both ugly, but at least it get's me out of a tight spot.
Matt
I get the sense this issue has multiple causes. I tried most of the solutions here. I cannot change our build server to use a PS script so that solution was out. Nothing I could try worked.
Finally, I deleted my solution and started a new one. The new solution worked. After diffing the broken solution with the working solution, I found the original solution was was missing lines. Each dependency that would not compile was missing this line:
{A93FB559-F0DB-4F4D-9569-E676F59D6168}.Release|Any CPU.Build.0 = Release|Any CPU
Note 1: The GUID will change from dependency to dependency.
Note 2: You can find lines like this one under the "GlobalSection(ProjectConfigurationPlatforms) = postSolution" part of the solution file.
My build.proj file says build "Release" using the "Any CPU" platform. Because MSBuild could not find this line it did not build this dependency. This results in the "error CS0246: The type or namespace could not be found" message.
If you are curious, someone had set this solution to the "x86" platform (which is wrong for us). I changed it to "Any CPU" (along with several other changes). Visual Studio did not add the corresponding lines to the solution file. Everything built fine in the IDE, but MSBuild started throwing errors.
I found that MSBuild is building projects from a solution in order in which they are declared in the .sln file. So if you reorder them with text editor you can fix the order for MSBuild.