Zextras 2.6.2 seems to be leaking memory rather badly.
I have upgraded to 8.8.6 GA from 8.7.11 and am still seeing the same gradual march towards Out of Memory crash. Running on Ubuntu 14.04.5 LTS
The server I am running on has 16GB of RAM and ~13 users active on ZxMobile Activesync. Its not really got that much going on but its gobbling RAM.
Increasing the mailbox heap size has reduced the crashes from 3-4 times per day to 1 per day.
Current settings are:
mailboxd_java_heap_new_size_percent = 30
mailboxd_java_heap_size = 4096
mailboxd_java_options = -server -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2 -Djava.awt.headless=true -Dsun.net.inetaddr.ttl= -Dorg.apache.jasper.compiler.disablejsr199=true -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewRatio=2 -XX:PermSize=196m -XX:MaxPermSize=350m -XX:SoftRefLRUPolicyMSPerMB=1 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:-OmitStackTraceInFastThrow -Xloggc:/opt/zimbra/log/gc.log -XX:-UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=4096K -Djava.net.preferIPv4Stack=true
These are as per the ZImbra wiki page https://wiki.zimbra.com/wiki/Performance_Tuning_Guidelines_for_Large_Deployments
2018-01-23 10:02:40.346:WARN:oejh.HttpParser:qtp1068934215-118177: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@7bb80dd0{r=1,c=false,a=IDLE,uri=null}
2018-01-23 22:42:36.043:WARN:oejs.ServletHandler:ZeXtras Mobile: Error for /service/extension/zimbrasync
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.zextras.utils.FastBase64.encodeToStream(FastBase64.java:371)
at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifier$SimpleDataHandler.writeTo(MimeSimplifier.java:126)
at javax.mail.internet.PreencodedMimeBodyPart.writeTo(PreencodedMimeBodyPart.java:112)
at javax.mail.internet.MimeMultipart.writeTo(MimeMultipart.java:462)
at com.sun.mail.handlers.multipart_mixed.writeTo(multipart_mixed.java:103)
at javax.activation.ObjectDataContentHandler.writeTo(DataHandler.java:889)
at javax.activation.DataHandler.writeTo(DataHandler.java:317)
at javax.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1485)
at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1773)
at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1749)
at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifierWithFallback.fixRawMime(MimeSimplifierWithFallback.java:51)
at com.zextras.modules.mobile.v2.engine.mime.SimplifierProxyBodyContentEncoder.encodeContentProvider(SimplifierProxyBodyContentEncoder.java:93)
at com.zextras.modules.mobile.v2.as.encoders.commons.EmailApplicationDataEncoder.encode(EmailApplicationDataEncoder.java:112)
at com.zextras.modules.mobile.v2.as.encoders.commons.SyncEncoder.encode(SyncEncoder.java:173)
at com.zextras.modules.mobile.v2.as.encoders.commons.LoggerEncoder.encode(LoggerEncoder.java:111)
at com.zextras.modules.mobile.v2.as.events.AddNewEmailEvent.encode(AddNewEmailEvent.java:39)
at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItem(SyncItemEmitter.java:445)
at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItems(SyncItemEmitter.java:312)
at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitChanges(SyncItemEmitter.java:108)
at com.zextras.modules.mobile.v2.engine.actions.CollectionSyncAction.execute(CollectionSyncAction.java:121)
at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
at com.zextras.modules.mobile.v2.as.assembler.ProbeAction.execute(ProbeAction.java:46)
at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
at com.zextras.modules.mobile.v2.engine.actions.StartEndWrapperAction.execute(StartEndWrapperAction.java:31)
at com.zextras.modules.mobile.v2.engine.actions.CheckFolderHierarchyAction.execute(CheckFolderHierarchyAction.java:98)
at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execAction(InitialExecutor.java:268)
at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execute(InitialExecutor.java:211)
Exception in thread "threads.csv" java.lang.instrument ASSERTION FAILED : "!errorOutstanding" with message can't create name string at JPLISAgent.c line: 807
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "threads.csv"
Exception in thread "sql.csv" Exception in thread "ldap.csv" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Cron" java.lang.OutOfMemoryError: Java heap space
(more, but forum has size limit)
As mentioned in other threads, ZxMobile seems very CPU heavy. Reducing http threads & turning off interprocess security has helped as has upgrading from 8.7.11 to 8.8.6 - but for a dozen clients to being pushing the server so hard is frustrating.
i.e.
/opt/zimbra/bin/zmprov modifyConfig zimbraHttpNumThreads 100
/opt/zimbra/bin/zmlocalconfig -e zimbra_require_interprocess_security=0
2018-01-23 10:02:40.346:WARN:oejh.HttpParser:qtp1068934215-118177: parse exception: java.lang.IllegalStateException: too much data seeking EOF in CLOSE for HttpChannelOverHttp@7bb80dd0{r=1,c=false,a=IDLE,uri=null}
2018-01-23 22:42:36.043:WARN:oejs.ServletHandler:ZeXtras Mobile: Error for /service/extension/zimbrasync
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at com.zextras.utils.FastBase64.encodeToStream(FastBase64.java:371)
at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifier$SimpleDataHandler.writeTo(MimeSimplifier.java:126)
at javax.mail.internet.PreencodedMimeBodyPart.writeTo(PreencodedMimeBodyPart.java:112)
at javax.mail.internet.MimeMultipart.writeTo(MimeMultipart.java:462)
at com.sun.mail.handlers.multipart_mixed.writeTo(multipart_mixed.java:103)
at javax.activation.ObjectDataContentHandler.writeTo(DataHandler.java:889)
at javax.activation.DataHandler.writeTo(DataHandler.java:317)
at javax.mail.internet.MimeBodyPart.writeTo(MimeBodyPart.java:1485)
at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1773)
at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1749)
at com.zextras.modules.mobile.v2.engine.mime.MimeSimplifierWithFallback.fixRawMime(MimeSimplifierWithFallback.java:51)
at com.zextras.modules.mobile.v2.engine.mime.SimplifierProxyBodyContentEncoder.encodeContentProvider(SimplifierProxyBodyContentEncoder.java:93)
at com.zextras.modules.mobile.v2.as.encoders.commons.EmailApplicationDataEncoder.encode(EmailApplicationDataEncoder.java:112)
at com.zextras.modules.mobile.v2.as.encoders.commons.SyncEncoder.encode(SyncEncoder.java:173)
at com.zextras.modules.mobile.v2.as.encoders.commons.LoggerEncoder.encode(LoggerEncoder.java:111)
at com.zextras.modules.mobile.v2.as.events.AddNewEmailEvent.encode(AddNewEmailEvent.java:39)
at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItem(SyncItemEmitter.java:445)
at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitNewItems(SyncItemEmitter.java:312)
at com.zextras.modules.mobile.v2.engine.actions.utils.SyncItemEmitter.emitChanges(SyncItemEmitter.java:108)
at com.zextras.modules.mobile.v2.engine.actions.CollectionSyncAction.execute(CollectionSyncAction.java:121)
at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
at com.zextras.modules.mobile.v2.as.assembler.ProbeAction.execute(ProbeAction.java:46)
at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
at com.zextras.modules.mobile.v2.engine.actions.ActionAggregator.execute(ActionAggregator.java:66)
at com.zextras.modules.mobile.v2.engine.actions.StartEndWrapperAction.execute(StartEndWrapperAction.java:31)
at com.zextras.modules.mobile.v2.engine.actions.CheckFolderHierarchyAction.execute(CheckFolderHierarchyAction.java:98)
at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execAction(InitialExecutor.java:268)
at com.zextras.modules.mobile.v2.engine.executors.InitialExecutor.execute(InitialExecutor.java:211)
Exception in thread "threads.csv" java.lang.instrument ASSERTION FAILED : "!errorOutstanding" with message can't create name string at JPLISAgent.c line: 807
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "threads.csv"
Exception in thread "sql.csv" Exception in thread "ldap.csv" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Cron" java.lang.OutOfMemoryError: Java heap space
(more, but forum has size limit)
As mentioned in other threads, ZxMobile seems very CPU heavy. Reducing http threads & turning off interprocess security has helped as has upgrading from 8.7.11 to 8.8.6 - but for a dozen clients to being pushing the server so hard is frustrating.
i.e.
/opt/zimbra/bin/zmprov modifyConfig zimbraHttpNumThreads 100
/opt/zimbra/bin/zmlocalconfig -e zimbra_require_interprocess_security=0