We have an installation of a 3-node carbonio (Carbonio core, Workstream server and Video server) for about 700 users. Sometimes we get the following errors:

https://nextcloud.illumiasolutions.com/s/qepnGM7jQZYq5kz?dir=/&editing=false&openfile=true

https://nextcloud.illumiasolutions.com/s/S3GA5XSkTzNgcxb?dir=/&editing=false&openfile=true

We have also taken the SOAP error message:

<soap:Envelope xmlns:soap=" removed link "><soap:Body><soap:Fault><soap:Code><soap:Value>soap:Sender</soap:Value></soap:Code><soap:Reason><soap:Text>parse error: empty request payload</soap:Text></soap:Reason><soap:Detail><Error xmlns="urn:zimbra"><Code>service.PARSE_ERROR</Code><Trace>qtp348026124-4002792:1774243864501:5dc02d0cee36c8a1</Trace></Error></soap:Detail></soap:Fault></soap:Body></soap:Envelope>

Can anyone tell what could be the possible reason of it and what is the permanent solution?

Thanks,

Biswajit Sinha

Carbonio Upgrade – Issues Encountered and How They Were Fixed

During the upgrade process, several issues appeared across different services (nginx, configd, postfix, and SSL).
The following notes document the symptoms, root causes, and the fixes that worked.

1. Nginx Failed to Start (Invalid tcp_keepalive Directive)

What Happened

carbonio-nginx.service failed immediately on startup.

carbonio-nginx.service: Failed with result 'exit-code'

Root Cause

The upgrade inserted an invalid directive into the proxy configuration:

tcp_keepalive on;

This line appeared in:

 removed link 

around line 164.
Nginx rejected the configuration because this directive is not valid in that context.

Fix

Comment the line and reload nginx.

sudo sed -i 's/^ tcp_keepalive on;/ #tcp_keepalive on;/'  removed link 

sudo /opt/zextras/common/sbin/nginx -t -c  removed link 

sudo systemctl restart carbonio-nginx.service

Important Note

This file is generated automatically by zmproxyconf, so the issue may come back if the proxy configuration is regenerated.

2. Message Dispatcher Database Was Not Bootstrapped

What Happened

Running:

pending-setups -a

returned the error:

Carbonio Message Dispatcher database password not found

Fix

Bootstrap the dispatcher database manually using the PostgreSQL admin credentials.

sudo PGPASSWORD='<PG_ADMIN_PASSWORD>' carbonio-message-dispatcher-db-bootstrap <PG_ADMIN_USER> 127.0.0.1 5432

Then run pending setups again:

sudo pending-setups -a

When prompted, provide the Consul password, which can be found here:

/var/lib/service-discover/password

3. carbonio-configd Entered a Restart Loop

What Happened

carbonio-configd was constantly restarting. The logs showed:

ImportError: No module named util

Restart counter exceeded 2000 attempts.

Root Cause

The zmpython classpath was missing the directory containing required jars:

/opt/zextras/lib/jars/*

Specifically, proxyconfgen.jar could not be located.

Fix

Edit the zmpython classpath.

sudo sed -i 's|-classpath "/opt/zextras/jython/jars/\*:/opt/zextras/mailbox/jars/\*:/opt/zextras/conf"|-classpath "/opt/zextras/jython/jars/*:/opt/zextras/lib/jars/*:/opt/zextras/mailbox/jars/*:/opt/zextras/conf"|' /opt/zextras/bin/zmpython

Then restart the service.

sudo systemctl restart carbonio-configd.service

Warning

The file belongs to the carbonio-core package, so this modification may be overwritten during future upgrades.

4. Postfix Ports 587 and 465 Not Listening

After a reboot, SMTP submission ports were not active.

Fix

Restart the postfix service.

sudo systemctl restart carbonio-postfix.service

Verify ports:

sudo ss -tlnp | grep -E ":(25|587|465) "

Expected ports:

• 25 – SMTP

• 587 – Submission

• 465 – SMTPS

5. SSL Certificate Expired

The installed certificate had expired and needed renewal using Let's Encrypt.

Requirement

Port 80 must be reachable from the internet.
Ensure NAT or firewall rules allow HTTP access to the mail server.

Step 1 – Enable HTTP Mode for Proxy

sudo su - zextras -c "zmprov ms $(hostname -f) zimbraReverseProxyMailMode both"

sudo su - zextras -c "zmproxyconf"

If the tcp_keepalive bug appears again, comment it and restart nginx.

Step 2 – Renew the Certificate

sudo /opt/zextras/common/bin/certbot renew --webroot -w /opt/zextras

Step 3 – Build the Full CA Chain

sudo curl -s -o  removed link   removed link 

sudo cat \
/opt/zextras/common/certbot/etc/letsencrypt/live/<YOUR_DOMAIN>/chain.pem \
 removed link  >  removed link 

Step 4 – Deploy the Certificate

sudo cp /opt/zextras/common/certbot/etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem \
 removed link 

sudo chown zextras:zextras  removed link 
sudo chmod 640  removed link 

sudo su - zextras -c "zmcertmgr deploycrt comm \
/opt/zextras/common/certbot/etc/letsencrypt/live/<YOUR_DOMAIN>/cert.pem \
 removed link "

Step 5 – Restart Services and Verify

Restart the main services.

sudo systemctl restart \
carbonio-nginx.service \
carbonio-appserver.service \
carbonio-postfix.service \
carbonio-openldap.service

Reload nginx.

sudo /opt/zextras/common/sbin/nginx -s reload

Verify the certificate:

echo | openssl s_client -connect 127.0.0.1:443 -servername <YOUR_MAIL_HOSTNAME> 2>/dev/null | openssl x509 -noout -dates

Check deployed certificates:

sudo su - zextras -c "zmcertmgr viewdeployedcrt"

Database Migration Verification

MySQL

db.version=111
index.version=2
redolog.version=1.42

These values match what is expected for Carbonio 26.3.x.

PostgreSQL

All expected databases are present:

• abq

• activesync

• auth

• backup

• carbonio-files-db

• carbonio-message-dispatcher-db

• carbonio_adm

• core

• ha

• powerstore

• others

Pending Setups

No remaining items.

Required DNS Records

Useful Commands (Quick Reference)

List Carbonio Services

sudo systemctl list-units --type=service | grep carbonio

Test Nginx Configuration

sudo /opt/zextras/common/sbin/nginx -t -c  removed link 

View Installed Certificates

sudo su - zextras -c "zmcertmgr viewdeployedcrt"

Check Database Version

sudo su - zextras -c "mysql -u root -p'PASS' -e 'SELECT * FROM zimbra.config'"

Check Pending Setups

sudo ls pending-setups.d | grep -v done

View Consul Password

sudo cat /var/lib/service-discover/password

Renew SSL

sudo /opt/zextras/common/bin/certbot renew --webroot -w /opt/zextras

Check Carbonio Version

sudo su - zextras -c "zmcontrol -v"

Check Mail Queue

sudo su - zextras -c "postqueue -p"

Note:
All domain names, hostnames, IP addresses, and passwords in this document use placeholders. Replace them with the real values before running the commands.

When checking the Carbonio service at OS level:

sudo systemctl status carbonio

The output is:

× carbonio.service - LSB: Carbonio Services
     Loaded: loaded (/etc/init.d/carbonio; generated)
     Active: failed (Result: exit-code) since Tue 2026-02-24 16:48:27 -06
       Docs: man:systemd-sysv-generator(8)
    Process: 801 ExecStart=/etc/init.d/carbonio start (code=exited, status=1/FAILURE)

Feb 24 16:46:11 systemd: Starting LSB: Carbonio Services...
Feb 24 16:46:15 su: (to zextras) root on none
Feb 24 16:46:16 pam_unix(su-l:session): session opened for user zextras(uid=998) by (uid=0)
Feb 24 16:48:27 systemd: carbonio.service: Control process exited, code=exited, status=1/FAILURE
Feb 24 16:48:27 systemd: carbonio.service: Failed with result 'exit-code'.
Feb 24 16:48:27 systemd: Failed to start LSB: Carbonio Services.

This indicates that the main carbonio.service fails when managed through systemd.

zmcontrol Status

Despite the above failure, when executing:

zmcontrol status

Most services appear operational, but we observe:

service-discover        Stopped

All other core services (amavis, antispam, antivirus, directory-server, mailbox, memcached, mta, opendkim, proxy, stats, config service) report as Running., i can sed mails others domains (gmail, outlook) but not send.