Opengrok OutOfMemoryError when re-indexing - reindex

Question

I have checked out from SVN 19 projects, all are in code source directory.
I run indexing from jenkins with following command:
C:\Jenkins\workspace\Grok-Multiple-Projects-Checkout-And-Indexing>java -Xmx12288m -Xms2048m -jar C:\grok_0.12.1\opengrok-0.12.1.5\lib\opengrok.jar -W C:\grok_0.12.1\Data\configuration.xml -c C:\grok_0.12.1\ctags58\ctags.exe -P -S -v -s C:\grok_0.12.1\src -d C:\grok_0.12.1\Data -i *.zip -i *.tmp -i *.db -i *.jar -i d:.svn -G -L polished -a on -T 8
First time I run indexing with the upper command: there are no errors!
However, consecutive runs will produce a
Java.lang.OutOfMemoryError: Java heap space
It runs fine until a point in the logs where it hangs for aprox 30 mins, and at some point memory consumption increases until it eats up all the allocated 12GB of RAM.
Here is the log:
09:38:40 Nov 01, 2016 9:38:45 AM org.opensolaris.opengrok.index.IndexDatabase$1 run
09:38:40 SEVERE: Problem updating lucene index database:
09:38:40 java.lang.OutOfMemoryError: Java heap space
09:38:40
09:38:41 Nov 01, 2016 9:38:45 AM org.opensolaris.opengrok.util.Statistics report
09:38:41 INFO: Done indexing data of all repositories (took 0:37:20)
09:38:41 Nov 01, 2016 9:38:45 AM org.opensolaris.opengrok.util.Statistics report
09:38:41 INFO: Total time: 0:37:21
09:38:41 Nov 01, 2016 9:38:45 AM org.opensolaris.opengrok.util.Statistics report
09:38:41 INFO: Final Memory: 19M/11,332M
Any ideas to why it needs so much memory and if increasing it will solve the OOM error. Could it be a memory leak in opengrok?

Answer 1

I know this is old question for old OpenGrok version (I can tell that because OpenGrok dropped the org.opensolaris class prefix in 2018), however I think it still needs some answer.
Assuming the indexer was indeed performing incremental reindex, there has to be something that is consuming the heap excessively. For instance, it could be caused by merging the pre existing history with the newly added (incremental) history.
Running the indexer with the -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/jvm/ Java options will create a heap dump that can be then analyzed with tools such as MAT (Eclipse) or YourKit.
As for memory leaks: not that it is not possible to create memory leaks in Java (via thread local storage), it is just quite improbable for them to manifest in the indexer.