Mailbox service is not starting after OS update this morning

I'm afraid that I'm going to be a 'me too', I upgraded earlier and encountered a failed server that wouldn't restart and it had the following in mailboxd.out:

javax.net.ssl|DEBUG|6F 13|qtp93314457-28435|2022-06-15 24:00:04.154 UTC|SSLExtensions.java:132|Ignore unknown or unsupported extension (
"unknown extension (45)": {
0000: 01 01 ..
}
)
javax.net.ssl|DEBUG|6F 13|qtp93314457-28435|2022-06-15 24:00:04.154 UTC|SSLExtensions.java:132|Ignore unknown or unsupported extension (
"unknown extension (51)": {
0000: 00 24 00 1D 00 20 72 6A 1F 18 38 98 2E CA F0 9C .$... rj..8.....
0010: EB 69 15 54 3F 6E 80 37 B1 4F 1E C6 AB F2 E0 40 .i.T?n.7.O.....@
0020: D9 1C A5 00 25 5E ....%^
}
)
javax.net.ssl|DEBUG|6F 13|qtp93314457-28435|2022-06-15 24:00:04.155 UTC|ClientHello.java:808|Consuming ClientHello handshake message (
"ClientHello": {
"client version" : "TLSv1.2",
"random" : "F7 28 4E 26 D4 AB 0C D2 CD D9 69 95 D7 9A B4 9A 48 77 F5 DF 6E D8 1B 51 A3 AC 85 30 5C 8F FD FC",
"session id" : "C2 79 29 CE 36 4F 86 20 06 E5 53 6F 81 DF 41 99 15 4B 16 A3 C2 00 95 84 E6 97 49 7A 1D 55 A2 B4",
"cipher suites" : "[TLS_AES_256_GCM_SHA384(0x1302), TLS_AES_128_GCM_SHA256(0x1301), TLS_CHACHA20_POLY1305_SHA256(0x1303), TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA2
56(0xC02B), TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA9), TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384(0xC030), TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA8), TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(0xC02F), TL
S_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384(0xC024), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384(0xC028), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256(0xC023), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(0xC027), TLS_ECDH_ECDSA_WITH_AES_25
6_GCM_SHA384(0xC02E), TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384(0xC032), TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256(0xC02D), TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256(0xC031), TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384(0xC026), TLS_EC
DH_RSA_WITH_AES_256_CBC_SHA384(0xC02A), TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256(0xC025), TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256(0xC029), TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA(0xC00A), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA(0
xC014), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA(0xC009), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA(0xC013), TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA(0xC005), TLS_ECDH_RSA_WITH_AES_256_CBC_SHA(0xC00F), TLS_ECDH_ECDSA_WITH_AES_128_CB
C_SHA(0xC004), TLS_ECDH_RSA_WITH_AES_128_CBC_SHA(0xC00E), TLS_RSA_WITH_AES_256_GCM_SHA384(0x009D), TLS_RSA_WITH_AES_128_GCM_SHA256(0x009C), TLS_RSA_WITH_AES_256_CBC_SHA256(0x003D), TLS_RSA_WITH_AES_128_CBC_SHA256(0x0
03C), TLS_RSA_WITH_AES_256_CBC_SHA(0x0035), TLS_RSA_WITH_AES_128_CBC_SHA(0x002F), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",
"compression methods" : "00",
"extensions" : [
"status_request (5)": {
"certificate status type": ocsp
"OCSP status request": {
"responder_id": <empty>
"request extensions": {
<empty>
}
}
},

I do have the full logfile if anyone needs to look at it or needs any further info from it.

Same. After updating some zimbra packages today, zimbra stopped working and java processes are dying with:

OpenJDK 64-Bit Server VM warning: .hotspot_compiler file is present but has been ignored.  Run with -XX:CompileCommandFile=.hotspot_compiler to load the file.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See  http://www.slf4j.org/codes.html#StaticLoggerBinder  for further details.
JettyMonitor monitoring thread pool QueuedThreadPool[qtp231977479]@dd3b207{STOPPED,10<=0<=250,i=0,r=-1,q=0}[NO_TRY]
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
        at org.eclipse.jetty.start.Main.start(Main.java:491)
        at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.security.PrivilegedActionException: java.lang.ClassNotFoundException: org.eclipse.jetty.setuid.SetUIDListener
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:573)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1857)
        ... 7 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.setuid.SetUIDListener
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        at org.eclipse.jetty.util.Loader.loadClass(Loader.java:64)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:1006)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.itemValue(XmlConfiguration.java:1540)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.value(XmlConfiguration.java:1441)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.access$700(XmlConfiguration.java:395)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$Args.<init>(XmlConfiguration.java:1699)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$Args.<init>(XmlConfiguration.java:1686)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.call(XmlConfiguration.java:942)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:515)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:468)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:380)
        at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$3(XmlConfiguration.java:1893)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        ... 8 more
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
        at org.eclipse.jetty.start.Main.start(Main.java:491)
        at org.eclipse.jetty.start.Main.main(Main.java:77)
Caused by: java.security.PrivilegedActionException: java.lang.ClassNotFoundException: org.eclipse.jetty.setuid.SetUIDListener
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:573)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1857)
        ... 7 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.setuid.SetUIDListener
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        at org.eclipse.jetty.util.Loader.loadClass(Loader.java:64)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:1006)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.itemValue(XmlConfiguration.java:1540)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.value(XmlConfiguration.java:1441)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.access$700(XmlConfiguration.java:395)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$Args.<init>(XmlConfiguration.java:1699)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$Args.<init>(XmlConfiguration.java:1686)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.call(XmlConfiguration.java:942)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:515)
        at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:468)
        at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:380)
        at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$3(XmlConfiguration.java:1893)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        ... 8 more

Usage: java -jar $JETTY_HOME/start.jar [options] [properties] [configs]
       java -jar $JETTY_HOME/start.jar --help  # for more information

I still can't figure out whats going on for the service not starting ...

But I didn't see those msgs on the log ...

Also, I only updated to the P24 after the service not starting.

The issue surfed after doing an 'apt upgrade' to the latest OS / Zimbra patches.

There is a thread on the Zimbra forums that also have different errors and mine was also an 'update' from the repo. My server is a CentOS 7.9 and the Zimbra version is:

zmcontrol -v
Release 9.0.0_ZEXTRAS_20220402.RHEL7_64_20220419075956 RHEL7_64 FOSS edition.

#me too.... Update from apt-get and mailboxd + webapps are not starting. Looks like Java issue.

I reverted also to the Zimbra forum. Please help. I can't find a solution currently.

I have the fully patched Ubuntu18.04 + Release 9.0.0.ZEXTRAS.20220402.UBUNTU18.64 UBUNTU18_64 FOSS + Patch v9.0P25

https://forums.zimbra.org/viewtopic.php?f=13&t=70843&p=305257#p305257  

Same here, Ubuntu 18.04 and Release 9.0.0.ZEXTRAS.20220402.UBUNTU18.64 UBUNTU18_64 FOSS edition 😥
Anyone having an idea how to emergency-solve this situation? Like apt-remove zimbra-* and reinstall without the zimbra repository?

I could bite myself for not saving a snapshot before...

I'm too tired for doing this ...

This is a solution on Zimbra Forum post above ...

Can someone give us a way of doing this on Ubuntu 18.04 ??

THIS STEPS ARE FOR CENTOS 7!

Remove:
zimbra-core-components.x86_64 0:3.0.12-1zimbra8.8b1.el7
zimbra-jetty-distribution.x86_64 0:9.4.46.v20220331-2.r7
zimbra-ldap-components.x86_64 0:2.0.6-1zimbra8.8b1.el7
zimbra-openjdk.x86_64 0:17.0.2-1zimbra8.8b1.el7

Install:
zimbra-core-components.x86_64 0:3.0.11-1zimbra8.8b1.el7
zimbra-jetty-distribution.x86_64 0:9.4.18.v20190429-2.r7
zimbra-ldap-components.x86_64 0:2.0.5-1zimbra8.8b1.el7
zimbra-openjdk.x86_64 0:13.0.1-1zimbra8.8b1.el7

If I try to remove one it gives a lot of dependencies also to be removed, and maybe someone more 'fresh mind' can solve this for us, also we need to be able to fetch the packages from some old repo, since the ones are the referenced as to be removed.

Trying to remove openjdk::

root@mail:~# apt-get remove zimbra-openjdk
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libasound2 libasound2-data libsensors4 libxi6 libxtst6 pax sysstat x11-common zimbra-amavis-logwatch zimbra-bdb-lib zimbra-curl zimbra-curl-lib zimbra-cyrus-sasl zimbra-freetype-lib
zimbra-ldap-base zimbra-libart-lib zimbra-libltdl-lib zimbra-libpng-lib zimbra-libsodium-lib zimbra-lmdb zimbra-openldap-client zimbra-openldap-server zimbra-openssl
zimbra-os-requirements zimbra-osl zimbra-perl zimbra-perl-archive-zip zimbra-perl-berkeleydb zimbra-perl-bit-vector zimbra-perl-canary-stability zimbra-perl-carp-clan
zimbra-perl-class-inspector zimbra-perl-convert-asn1 zimbra-perl-convert-binhex zimbra-perl-convert-tnef zimbra-perl-convert-uulib zimbra-perl-crypt-openssl-random
zimbra-perl-crypt-saltedhash zimbra-perl-data-uuid zimbra-perl-date-calc zimbra-perl-date-manip zimbra-perl-db-file zimbra-perl-dbd-mysql zimbra-perl-email-date-format
zimbra-perl-encode-locale zimbra-perl-file-grep zimbra-perl-file-libmagic zimbra-perl-file-listing zimbra-perl-file-tail zimbra-perl-filesys-df zimbra-perl-geography-countries
zimbra-perl-http-cookies zimbra-perl-http-daemon zimbra-perl-http-date zimbra-perl-http-message zimbra-perl-http-negotiate zimbra-perl-innotop zimbra-perl-io-html
zimbra-perl-io-sessiondata zimbra-perl-io-socket-ip zimbra-perl-io-socket-ssl zimbra-perl-io-stringy zimbra-perl-ip-country zimbra-perl-json-pp zimbra-perl-libwww
zimbra-perl-lwp-mediatypes zimbra-perl-lwp-protocol-https zimbra-perl-math-bigint zimbra-perl-mime-lite zimbra-perl-mime-tools zimbra-perl-mime-types zimbra-perl-mozilla-ca
zimbra-perl-net-http zimbra-perl-net-ldap zimbra-perl-net-ldapapi zimbra-perl-net-libidn zimbra-perl-net-ssleay zimbra-perl-parent zimbra-perl-proc-processtable zimbra-perl-soap-lite
zimbra-perl-socket-linux zimbra-perl-swatchdog zimbra-perl-task-weaken zimbra-perl-term-readkey zimbra-perl-unix-getrusage zimbra-perl-unix-syslog zimbra-perl-uri
zimbra-perl-www-robotrules zimbra-perl-xml-namespacesupport zimbra-perl-xml-parser zimbra-perl-xml-parser-lite zimbra-perl-xml-sax zimbra-perl-xml-sax-base zimbra-perl-xml-sax-expat
zimbra-perl-xml-simple zimbra-perl-zmq-constants zimbra-perl-zmq-libzmq3 zimbra-pflogsumm zimbra-postfix-logwatch zimbra-prepflog zimbra-rrdtool zimbra-rrdtool-lib zimbra-rsync
zimbra-tcmalloc-lib zimbra-zeromq-lib
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
zimbra-apache zimbra-chat zimbra-common-core-jar zimbra-common-core-libs zimbra-common-mbox-conf zimbra-common-mbox-conf-attrs zimbra-common-mbox-conf-msgs zimbra-common-mbox-conf-rights
zimbra-common-mbox-db zimbra-common-mbox-docs zimbra-common-mbox-native-lib zimbra-core zimbra-core-components zimbra-dnscache zimbra-drive zimbra-ldap zimbra-ldap-components
zimbra-logger zimbra-mbox-service zimbra-mta zimbra-openjdk zimbra-openjdk-cacerts zimbra-proxy zimbra-snmp zimbra-spell zimbra-store zimbra-timezone-data
0 upgraded, 0 newly installed, 27 to remove and 24 not upgraded.
After this operation, 434 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.

Ex:

root@mail:~# apt search zimbra-openjdk
Sorting... Done
Full Text Search... Done
zimbra-openjdk/unknown,unknown,now 17.0.2-1zimbra8.8b1.18.04 amd64 [installed,automatic]
OpenJDK

zimbra-openjdk-cacerts/unknown,unknown,now 1.0.8-1zimbra8.7b1.18.04 all [installed,automatic]
CA Certs keystore for OpenJDK

zimbra-openjdk-dbg/unknown,unknown 17.0.2-1zimbra8.8b1.18.04 amd64
OpenJDK binary debug information

Thanks in advance

JG

Has anyone logged a ticket to support? Just asking because I am an private user and use OSE.... 🙂 And a happy user of Zimbra based on Zextras.

Posted by: @martinwiertz

Has anyone logged a ticket to support? Just asking because I am an private user and use OSE.... 🙂 And a happy user of Zimbra based on Zextras.

That would require an NE customer to report a problem with the OSE version of Zimbra, I'd bet my last dollar that Zimbra isn't interested in following up the problem in these circumstances but I'd love to be proved wrong. 🙂 

@phoenix NE version has no problems?? Usually these problems are common.

Posted by: @phoenix

Posted by: @martinwiertz

Has anyone logged a ticket to support? Just asking because I am an private user and use OSE.... 🙂 And a happy user of Zimbra based on Zextras.

That would require an NE customer to report a problem with the OSE version of Zimbra, I'd bet my last dollar that Zimbra isn't interested in following up the problem in these circumstances but I'd love to be proved wrong. 🙂 

 

Agree. Since they moved to 'paid' solutions ... all is a mess.

Zextras is doing a great job, but ... we keep being dependent of Zimbra ...

I have to consider either other options (MIAB) is a good simple mail server or Zextras Carbonio, but since it's also Zimbra depended, still ... don't know yet what to do ...

I might fall for Carbonio to support Zextras efforts ...

Can someone be kind enough to write a small how-to to remove and install those components??

I can't, I m too damm tired of this ...

I have my restored server on standby until tomorrow 5 am WEST.

After that I start it and open my firewall for incoming mails ... but this one has almost 7 hrs of emails in it ...

Let's hope for the best.

It would be great if someone from Zextras could step in and let us know if they have any insights into this? We are struggling to get a safe route out. Zimbra dont support OSE 9 in any form only OSE 8.8x which looks like it now is being left without some new fixes needed. Carbonio would perhaps work but not quite yet from a frying pan and fire analogy.

NE versions seem to be updating fine. Certainly feels like we have fewer places to turn - but Zextras probably have similar update issues with Carbonio elements so could be well palced to help. At present I would be tempted by a migration option! 

I don't know when this post will be available because I just registered this forum account and all my posts are in state "Awaiting moderation"

Anyway, for those of you who are using Ubuntu 18.04 LTS with zimbra 9.0.0.ZEXTRAS.20220402.UBUNTU18.64) this package downgrade will fix it:

apt-get install zimbra-openjdk=13.0.1-1zimbra8.8b1.18.04 zimbra-core-components=3.0.11-1zimbra8.8b1.18.04 zimbra-jetty-distribution=9.4.18.v20190429-2.u18 zimbra-ldap-components=2.0.5-1zimbra8.8b1.18.04