Cloudera Manager Embedded PostgreSQL Hive Metastore Server OutOfMemoryError Issue - hive

Question

I'm using:
Cloudera Manager Free Edition: 4.5.1
Cloudera Hadoop Distro: CDH 4.2.0-1.cdh4.2.0.p0.10 (Parcel)
Hive Metastore with cloudera manager embedded PostgreSQL database.
My cloudera manager is running on a separate machine and it's not part of the cluster.
After setting up the cluster using cloudera manager, I started using hive through hue + beeswax.
Everything was running fine for a while and then all of a suddden, whenever I ran any query against a particular table that had a large number of partitions (about 14000), the query started to time out:
FAILED: SemanticException org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
When I noticed this, I looked at the logs and found out that the connection to the Hive Metastore was timing out:
WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
Having seen this, I thought there was a problem with the hive metastore. So I looked at the logs for the hive metastore and discovered java.lang.OutOfMemoryErrors:
/var/log/hive/hadoop-cmf-hive1-HIVEMETASTORE-hci-cdh01.hcinsight.net.log.out:
2013-05-07 14:13:08,744 ERROR org.apache.thrift.ProcessFunction: Internal error processing get_partitions_ with_auth
java.lang.OutOfMemoryError: Java heap space
at sun.reflectH.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.jav a:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.datanucleus.util.ClassUtils.newInstance(ClassUtils.java:95)
at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newLiteralParameter(SQLExpressi onFactory.java:248)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapE ntrySetStore.java:323)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java: 221)
at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987)
at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258)
at org.datanucleus.sco.backed.Map.keySet(Map.java:509)
at org.datanucleus.store.fieldmanager.LoadFieldManager.internalFetchObjectField(LoadFieldManager.j ava:118)
at org.datanucleus.store.fieldmanager.AbstractFetchFieldManager.fetchObjectField(AbstractFetchFiel dManager.java:114)
at org.datanucleus.state.AbstractStateManager.replacingObjectField(AbstractStateManager.java:1183)
at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.ja va)
at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceFields(MStorageDescriptor.j ava)
at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2860)
at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2879)
at org.datanucleus.jdo.state.JDOStateManagerImpl.loadFieldsInFetchPlan(JDOStateManagerImpl.java:16 47)
at org.datanucleus.store.fieldmanager.LoadFieldManager.processPersistable(LoadFieldManager.java:63 )
at org.datanucleus.store.fieldmanager.LoadFieldManager.internalFetchObjectField(LoadFieldManager.j ava:84)
at org.datanucleus.store.fieldmanager.AbstractFetchFieldManager.fetchObjectField(AbstractFetchFiel dManager.java:104)
at org.datanucleus.state.AbstractStateManager.replacingObjectField(AbstractStateManager.java:1183)
at org.apache.hadoop.hive.metastore.model.MPartition.jdoReplaceField(MPartition.java)
at org.apache.hadoop.hive.metastore.model.MPartition.jdoReplaceFields(MPartition.java)
at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2860)
at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2879)
at org.datanucleus.jdo.state.JDOStateManagerImpl.loadFieldsInFetchPlan(JDOStateManagerImpl.java:16 47)
at org.datanucleus.ObjectManagerImpl.performDetachAllOnTxnEndPreparation(ObjectManagerImpl.java:35 52)
at org.datanucleus.ObjectManagerImpl.preCommit(ObjectManagerImpl.java:3291)
at org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:369)
at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:256)
At this point, the hive metastore gets shutdown and restarted:
2013-05-07 14:39:40,576 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Shutting down hive metastore.
2013-05-07 14:41:09,979 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Starting hive metastore on po rt 9083
Now, to fix this, I've changed the max heap size of both the hive metastore server and the beeswax server:
1. Hive/Hive Metastore Server(Base)/Resource Management/Java Heap Size of Metastore Server : 2 GiB (First thing I did.)
2. Hue/Beeswax Server(Base)/Resource Management/Java Heap Size of Beeswax Server : 2 GiB (After reading some groups posts and stuff online, I tried this as well.)
Neither of these above 2 steps seem to have helped as I continue to see OOMEs in the hive metastore log.
Then I noticed that the actual metastore 'database' is being run as part of my cloudera manager and I'm wondering if that PostgreSQL process is running out of memory. I looked for ways to increase the java heap size for that process and found very little documentation regarding that.
I was wondering if one of you guys could help me solve this issue.
Should I increase the java heap size for the embedded database? If so, where would I do this?
Is there something else that I'm missing?
Thanks!

Answer 1

Did you try doing the below.
'SET hive.metastore.client.socket.timeout=300;'
This solved the issue for me. Let me know how it went.