Apache Ignite: Long GC pause on Cross Join of two tables - jvm

Question

I've setup of 4 node Ignite cluster with following JVM flags.
ENV JAVA_OPTS="-Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -server -Xms1g\
-Xmx1500m -XX:+AlwaysPreTouch -XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled \
-XX:+CMSPermGenSweepingEnabled -XX:+CMSScavengeBeforeRemark \
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC \
-XX:+UseContainerSupport \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation\
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xloggc:/opt/ignite/logs/log.txt"
I tried with both G1 and CMS garbage collector. And limited memory to 1500mb and minimum of 1gb. I've attached the logs.
One more thing I've added +PrintGCDetails and have provided log file path to dump the GC logs but file isn't getting generated(I've the folder logs). what is the issue?
for reference both tables have 100k rows.
Logs file

Answer 1

In the log, I see
[09:36:45,955][WARNING][long-qry-#31][LongRunningQueryManager] Query execution is too long [globalQueryId=2e3ce071-b1cd-48d9-bb4c-22ec428a8e19_226, duration=48660ms, type=MAP, distributedJoin=false, enforceJoinOrder=false, lazy=false, schema=PUBLIC, sql='SELECT
A__Z0.FIRST_NAME __C0_0,
B__Z1.LAST_NAME __C0_1,
PUBLIC.LEVENSHTEINDISTANCE(A__Z0.FIRST_NAME, B__Z1.LAST_NAME) __C0_2
FROM PUBLIC.LEADS A__Z0
INNER JOIN PUBLIC.CONTACTS B__Z1
ON TRUE
WHERE TRUE', plan=SELECT
A__Z0.FIRST_NAME AS __C0_0,
B__Z1.LAST_NAME AS __C0_1,
PUBLIC.LEVENSHTEINDISTANCE(A__Z0.FIRST_NAME, B__Z1.LAST_NAME) AS __C0_2
FROM PUBLIC.LEADS A__Z0
/* PUBLIC.LEADS.__SCAN_ */
/* WHERE TRUE
*/
/* scanCount: 163 */
INNER JOIN PUBLIC.CONTACTS B__Z1
/* PUBLIC.CONTACTS.__SCAN_ */
ON 1=1
/* scanCount: 4082223 */
WHERE TRUE, reqId=17, segment=0]
Note the lazy=false. What it tells me is that Ignite will try to collect the entire cross join product in the memory and then serve it to the client. This is why you run out of memory.
Set lazy=true to this query (actually, set it by default to most your SQL queries).
See this for details.