Hi,
We recently upgraded from Zextras Carbonio CE from 23.12 to 24.1, post which barring 2 domains, on our Carbonio CE server the third and fourth stopped working. When the user logged in, it started showing "Some technical issues occurred while processing your request. Please try to refresh the page or go back to login page". All domains were working before this upgrade. Now I am not even able to delete the non-working domain and its users.
Can some one from community help us?
Best regards,
Raghav
@a_raghava
Hi,
Sorry to hear about the trouble.
I just checked by upgrading from 23.12.0 to 24.1.0 with 3 domains. All of the are working and I simply followed the official doc https://docs.zextras.com/carbonio-ce/html/release/upgrade.html
I even deleted one of them.
What OS you are using?
Also share the output below commands:
systemctl list-units carbonio* consul members systemctl status service-discover.service zmcontrol status
Regards,
Sharif
Hi,
Thanks for your quick response.
We also upgraded following the same procedure given the same link. Upgrade process went through smoothly. There were no issues.
But when the users of the one of our test domains, tried to login they started seeing the error message shown in the previous message above.
OS : Ubuntu 20.04.06 LTS
Coming to outputs of the command you have given:
1. systemctl list-units carbonio*
loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
2. consul members
Node Address Status Type Build Protocol DC Segment
server-mail-exzatech-net 172.31.xx.xxx:8301 alive server 1.10.12 2 dc1 <all>
3. systemctl status service-discover.service
Loaded: loaded (/lib/systemd/system/service-discover.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-02-13 10:31:58 UTC; 22h ago
Docs: https://docs.zextras.com/
Main PID: 1949 (consul)
asks: 12 (limit: 18945)
Memory: 87.1M
CGroup: /system.slice/service-discover.service
└─1949 /usr/bin/consul agent -bootstrap-expect 1 -config-dir /etc/zextras/service-discover/ -server -retry-join=mail.exzatech.net
4. zmcontrol status
amavis Running
antispam Running
antivirus Running
directory-server Running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
service-discover Running
stats Running
config service Running
Hope this helps in identifying the issue.
Best regards,
Raghav
@a_raghava
Hi,
systemctl list-units carbonio* should show carbonio unit files like this:
root@mail:~# systemctl list-units carbonio* UNIT LOAD ACTIVE SUB DESCRIPTION carbonio-clamav-sidecar.service loaded active running Carbonio clamav sidecar proxy carbonio-clamav-signature-provider-sidecar.service loaded active running NGINX Carbonio ClamAV signature provider sidecar carbonio-docs-connector-db-sidecar.service loaded active running Carbonio Docs Connector database sidecar carbonio-docs-connector-sidecar.service loaded active running Carbonio docs connector sidecar proxy carbonio-docs-connector.service loaded active running Carbonio docs connector daemon carbonio-docs-editor-sidecar.service loaded active running Carbonio Docs server sidecar proxy carbonio-docs-editor.service loaded active running Carbonio Docs Server carbonio-files-db-sidecar.service loaded active running Carbonio Files database sidecar carbonio-files-sidecar.service loaded active running Carbonio Files sidecar proxy carbonio-files.service loaded active running Carbonio Files daemon carbonio-mailbox-db-sidecar.service loaded active running Carbonio Mailbox database sidecar carbonio-mailbox-sidecar.service loaded active running Mailbox carbonio proxy sidecar carbonio-message-broker-sidecar.service loaded active running Carbonio message broker sidecar proxy carbonio-message-broker.service loaded active running Carbonio message broker daemon carbonio-message-dispatcher-auth-sidecar.service loaded active running Carbonio Message Dispatcher auth sidecar proxy carbonio-message-dispatcher-auth.service loaded active running Carbonio Message Dispatcher auth daemon carbonio-message-dispatcher-db-sidecar.service loaded active running Carbonio Message Dispatcher database sidecar carbonio-message-dispatcher-http-sidecar.service loaded active running Carbonio Message Dispatcher HTTP sidecar proxy carbonio-message-dispatcher-xmpp-sidecar.service loaded active running Carbonio Message Dispatcher XMPP sidecar proxy carbonio-message-dispatcher.service loaded active running Carbonio Message Dispatcher daemon carbonio-mta-sidecar.service loaded active running Carbonio mta sidecar proxy carbonio-preview-sidecar.service loaded active running Carbonio Preview sidecar proxy carbonio-preview.service loaded active running Carbonio Preview daemon carbonio-prometheus-consul-exporter.service loaded active running Carbonio Prometheus Consul exporter carbonio-prometheus-mysqld-exporter.service loaded active running Carbonio Prometheus MySQL exporter carbonio-prometheus-nginx-exporter.service loaded active running Carbonio Prometheus NGINX exporter carbonio-prometheus-node-exporter.service loaded active running Carbonio Prometheus Node Exporter carbonio-prometheus-openldap-exporter.service loaded active running Carbonio Prometheus OpenLDAP exporter carbonio-proxy-sidecar.service loaded active running NGINX carbonio proxy sidecar carbonio-storages-sidecar.service loaded active running Storages-CE sidecar proxy carbonio-storages.service loaded active running Carbonio Storages-CE daemon carbonio-tasks-db-sidecar.service loaded active running Carbonio Tasks database sidecar carbonio-tasks-sidecar.service loaded active running Carbonio Tasks sidecar proxy carbonio-user-management-sidecar.service loaded active running Carbonio User Management sidecar proxy carbonio-user-management.service loaded active running Carbonio User Management daemon carbonio-videoserver-sidecar.service loaded active running Carbonio Videoserver sidecar proxy carbonio-videoserver.service loaded active running Zextras Videoserver carbonio-ws-collaboration-db-sidecar.service loaded active running Carbonio Workstream Collaboration database sidecar carbonio-ws-collaboration-sidecar.service loaded active running Carbonio Workstream Collaboration sidecar proxy carbonio-ws-collaboration.service loaded active running Carbonio Workstream Collaboration daemon ● carbonio.service loaded failed failed LSB: Carbonio Services LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 41 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. root@mail:~#
Try rebooting the server, then
systemctl restart carbonio-message-broker systemctl restart carbonio-videoserver systemctl restart carbonio-ws-collaboration systemctl restart carbonio-message-dispatcher systemctl restart carbonio-tasks.service systemctl restart carbonio-prometheus-nginx-exporter.service
Check again after all services get up and running:
zmcontrol status
Regards,
Sharif
Hi,
After rebooting the server and restarting all services as you suggested, systemctl list-unit carbonio* is showing all units.
However, I am still not able to delete the non functional domain. Force delete despite leaving for 30 minutes, there is no success. Only the account is getting closed (Soft Delete).
root@mail:/home/ubuntu# systemctl list-units carbonio* UNIT LOAD ACTIVE SUB DESCRIPTION carbonio-clamav-sidecar.service loaded active running Carbonio clamav sidecar proxy carbonio-clamav-signature-provider-sidecar.service loaded active running NGINX Carbonio ClamAV signature provider sidecar carbonio-docs-connector-sidecar.service loaded active running Carbonio docs connector sidecar proxy carbonio-docs-connector.service loaded active running Carbonio docs connector daemon carbonio-docs-editor-sidecar.service loaded active running Carbonio Docs server sidecar proxy carbonio-docs-editor.service loaded active running Carbonio Docs Server carbonio-files-db-sidecar.service loaded active running Carbonio Files database sidecar carbonio-files-sidecar.service loaded active running Carbonio Files sidecar proxy carbonio-files.service loaded active running Carbonio Files daemon carbonio-mailbox-sidecar.service loaded active running Mailbox carbonio proxy sidecar carbonio-mta-sidecar.service loaded active running Carbonio mta sidecar proxy carbonio-preview-sidecar.service loaded active running Carbonio Preview sidecar proxy carbonio-preview.service loaded active running Carbonio Preview daemon carbonio-prometheus-consul-exporter.service loaded active running Carbonio Prometheus Consul exporter carbonio-prometheus-mysqld-exporter.service loaded active running Carbonio Prometheus MySQL exporter carbonio-prometheus-nginx-exporter.service loaded active running Carbonio Prometheus NGINX exporter carbonio-prometheus-node-exporter.service loaded active running Carbonio Prometheus Node Exporter carbonio-prometheus-openldap-exporter.service loaded active running Carbonio Prometheus OpenLDAP exporter carbonio-proxy-sidecar.service loaded active running NGINX carbonio proxy sidecar carbonio-storages-sidecar.service loaded active running Storages-CE sidecar proxy carbonio-storages.service loaded active running Carbonio Storages-CE daemon carbonio-tasks-db-sidecar.service loaded active running Carbonio Tasks database sidecar carbonio-tasks-sidecar.service loaded active running Carbonio Tasks sidecar proxy carbonio-user-management-sidecar.service loaded active running Carbonio User Management sidecar proxy carbonio-user-management.service loaded active running Carbonio User Management daemon carbonio.service loaded active exited LSB: Carbonio Services LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 26 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'.
Best regards,
Raghav
@a_raghava
Could you please check mailbox.log or audit.log for any hints?
Thanks,
Sharif
Hi,
The mailbox.log has lot of entries like:
2024-02-14 11:08:51,595 ERROR [qtp532677950-376] [name=admin@xxx.net;mid=3;ip=172.31.47.165;oip=122.171.22.86;port=48544;ua=CarbonioWebClient - Firefox 122.0 (Ubuntu)/24.1.0_ZEXTRAS_202401 carbonio 20240104-1254 FOSS;soapId=7ead6577;] mailbox - Error during mailbox creation com.zimbra.common.service.ServiceException: system failure: unable to read SQL statements from /opt/zextras/mailboxd/../db/create_database.sql at com.zimbra.cs.mailbox.MailboxManager.createMailboxInternal(MailboxManager.java:905) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.createMailbox(MailboxManager.java:878) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccountId(MailboxManager.java:387) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccountId(MailboxManager.java:342) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccount(MailboxManager.java:279) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccount(MailboxManager.java:235) ~[zimbrastore.jar:?] Caused by: java.io.FileNotFoundException: /opt/zextras/mailboxd/../db/create_database.sql (No such file or directory) com.zimbra.common.service.ServiceException: system failure: unable to read SQL statements from /opt/zextras/mailboxd/../db/create_database.sql at com.zimbra.cs.mailbox.MailboxManager.createMailboxInternal(MailboxManager.java:905) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.createMailbox(MailboxManager.java:878) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccountId(MailboxManager.java:387) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccountId(MailboxManager.java:342) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccount(MailboxManager.java:279) ~[zimbrastore.jar:?] at com.zimbra.cs.mailbox.MailboxManager.getMailboxByAccount(MailboxManager.java:235) ~[zimbrastore.jar:?] Caused by: java.io.FileNotFoundException: /opt/zextras/mailboxd/../db/create_database.sql (No such file or directory) 2024-02-14 11:09:28,016 INFO [ImapServer-6] [name=admin@xxx.net;mid=3;ip=172.31.47.165;oip=122.171.22.86;via=172.31.47.165(nginx/1.20.2);ua=Zimbra/24.1.0_ZEXTRAS_202401;cid=8;] mailbox - total space required to copy items = 34342
And the audit.log has nothing except account addition, disable and close entries.
Looks like mailbox is not getting created due to some sql file non availability. I am not able to find this file anywhere in the whole server.
But can't still understand why the domain is not getting deleted??!!
Best regards,
Raghav