Glassfish/Payara: why might deployment of a large web app to Glassfish-4.1.1 (or Payara Server 4.1.1.163) take 5 times longer than to Glassfish-4.1 - glassfish-4.1

Question

Mac OS X: Yosemite 10.10.5
Until recently I have been using:
NetBeans8.1beta
Glassfish4.1
Mojarra 2.2.7
I have recently investigated using:
NetBeans8.1
Glassfish4.1.1
Mojarra 2.2.12.
With a very large JSF web app (with ObjectDB as JPA persistence provider) I am experiencing massively longer deployment times to Glassfish4.1.1 (over 10 minutes) than to Glassfish4.1 (around 2 minutes). That's about 5 times slower.
[EDIT: 2016-09-13 Tried with Payara 163 Full (Payara Server 4.1.1.163) and it took over 20 minutes, even slower than on Glassfish4.1.1, with nearly identical setup]
I can't possibly reproduce here or provide examples of the large web app.
Q1: Has anybody experienced any similar difference between Glassfish-4.1 and Glassfish-4.1.1 [EDIT: OR Payara Server 4.1.1.163], and where I might I look for the point of difference ?
(I am already trying to investigate it by using the NetBeans Profiler, but it is not turning out to be so easy yet to find the cause.)
[EDIT: 2016-09-15 timing tests]
I have performed careful side-by-side comparison using screencast recording and comparing with timings in the log.
In Payara41 (4.1.1.163) and Glassfish4.1.1 it gets "stuck" as shown below for about 10 minutes, and I have not found any log setting that gives a single line of output here where the problem occurs, and running the server in DEBUG mode also does not seem to give any additional information.
[2016-09-14T23:02:53.450+1000] [Payara 4.1] [INFO] [NCLS-LOGGING-00009] [javax.enterprise.logging] [tid: _ThreadID=19 _ThreadName=RunLevelControllerThread-1473858173343] [timeMillis: 1473858173450] [levelValue: 800] [[
Running Payara Version: Payara Server 4.1.1.163 #badassfish (build 215)]]
... runs ok through all steps and then through dozens of these EJB JNDI name declarations:
[2016-09-14T23:03:48.823+1000] [Payara 4.1] [INFO] [AS-EJB-00054] [javax.enterprise.ejb.container] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858228823] [levelValue: 800] [[
Portable JNDI names for EJB BeanName: [java:global/app-name/BeanName!com.example.BeanName, java:global/app-name/BeanName]]]
Gives this WELD version message:
[2016-09-14T23:03:48.875+1000] [Payara 4.1] [INFO] [] [org.jboss.weld.Version] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858228875] [levelValue: 800] [[
WELD-000900: 2.3.5 (Final)]]
... then takes about 8 seconds before this message ..
[2016-09-14T23:03:56.666+1000] [Payara 4.1] [INFO] [] [org.jboss.weld.Event] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858236666] [levelValue: 800] [[
WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(#Observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using #WithAnnotations or a generic type with bounds.]]
... Then take many minutes (typically about 10 minutes) before resuming these otherwise harmless warnings ..
[2016-09-14T23:11:33.387+1000] [Payara 4.1] [INFO] [] [org.jboss.weld.Bootstrap] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858693387] [levelValue: 800] [[
WELD-001125: Illegal bean type java.util.Comparator<javax.persistence.metamodel.EntityType<?>> ignored on [EnhancedAnnotatedTypeImpl] public static class com.example.ElementManager$EntityTypeSorter]]
[2016-09-14T23:11:59.736+1000] [Payara 4.1] [WARNING] [] [org.glassfish.jersey.internal.Errors] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858719736] [levelValue: 900] [[
The following warnings have been detected: WARNING: Parameter 1 of type T from public void com.example.NumberProperty.setDefaultValue(T) is not resolvable to a concrete type.
]]
Similar warning messages happen in the Glassfish4.1 log, but there is no long pause/holdup.
Q2: What else can I do to find out why it is pausing/stalling there for ages ?

Answer 1

Thanks to detailed help from the Payara team I have an answer.
The max heap settings were different between my old Glassfish4.1 install (-Xmx2048m) and the new Glassfish4.1.1 and Payara41 install (-Xmx512m). On building my web app project's database model on startup, it was hitting the 512m heap limit and causing excessive garbage collection (as revealed by JVisualVM), which massively slowed the deployment.
I already had these as run-time VMargs for the NetBeans web app project:
-Xms2048m -Xmx2048m
But on installing NetBeans8.1 (with Glassfish4.1.1), and then Payara41, I had neglected to also match this on the new servers to make room for it. With the increased max heap and fixed minimum the deployment time is now about 2 mins on both Glassfish4.1.1 and Payara41.