im on Ubuntu 22.04.5 LTS with Carbonio CE 24.12.1, single server
not sure what i did, but carbonio ce was running great until a few days ago when all my logs started to show the some kind of connection refusal to port 8500.
as a result none of my accounts are not able to send mail from the webmail, nor dedicated iphoneapp
but the email can be accessed via IMAP
i think it has something to do with the service-discover, where i found a few articles on here, and tried them all, but not able to "restart" it. all other services work fine.
sos - help pls..
@jajaish
Could you please share the output of following commands:
root@mail:~# su - zextras -c "zmcontrol status"
root@mail:~# systemctl list-units carbonio*
root@mail:~# curl -s -v http://127.78.0.4:10000/health | jq
Also, please feel free to take a look at this segment of documentation:
https://docs.zextras.com/carbonio-ce/html/troubleshooting/toc.html
Regards,
Hello jajaish,
You can either support restarting the discovery service right then and there, or run the screen with the command below to see what errors occurred and resend the error log so people can support you more easily.
journalctl -u service-discover -f
thx thx
su - zextras -c "zmcontrol status"
antivirus Running
directory-server Running
mailbox Running
memcached Running
mta Running
proxy Running
service webapp Running
service-discover Stopped
stats Running
config service Running
systemctl list-units carbonio*
carbonio-catalog-sidecar.service loaded activating auto-restart Carbonio Catalog sidecar proxy
carbonio-catalog.service loaded active running Carbonio Catalog daemon
carbonio-clamav-sidecar.service loaded activating auto-restart Carbonio clamav sidecar proxy
carbonio-clamav-signature-provider-sidecar.service loaded activating auto-restart NGINX Carbonio ClamAV signature provider si> carbonio-docs-connector-db-sidecar.service loaded activating auto-restart Carbonio Docs Connector database sidecar
carbonio-docs-connector-sidecar.service loaded activating auto-restart Carbonio docs connector sidecar proxy
carbonio-docs-connector.service loaded active running Carbonio docs connector daemon
carbonio-docs-editor-sidecar.service loaded activating auto-restart Carbonio Docs server sidecar proxy
carbonio-docs-editor.service loaded active running Carbonio Docs Server
carbonio-files-db-sidecar.service loaded activating auto-restart Carbonio Files database sidecar
carbonio-files-sidecar.service loaded activating auto-restart Carbonio Files sidecar proxy
carbonio-files.service loaded activating auto-restart Carbonio Files daemon
carbonio-mailbox-admin-sidecar.service loaded activating auto-restart Carbonio Mailbox Admin proxy sidecar
carbonio-mailbox-db-sidecar.service loaded activating auto-restart Carbonio Mailbox database sidecar
carbonio-mailbox-nslookup-sidecar.service loaded activating auto-restart Carbonio Mailbox NSLookup proxy sidecar
carbonio-mailbox-sidecar.service loaded activating auto-restart Mailbox carbonio proxy sidecar
carbonio-message-broker-sidecar.service loaded activating auto-restart Carbonio message broker sidecar proxy
carbonio-message-broker.service loaded active running Carbonio message broker daemon
carbonio-message-dispatcher-auth-sidecar.service loaded activating auto-restart Carbonio Message Dispatcher auth sidecar pr> carbonio-message-dispatcher-auth.service loaded active running Carbonio Message Dispatcher auth daemon
carbonio-message-dispatcher-db-sidecar.service loaded activating auto-restart Carbonio Message Dispatcher database sidecar carbonio-message-dispatcher-http-sidecar.service loaded activating auto-restart Carbonio Message Dispatcher HTTP sidecar pr> carbonio-message-dispatcher-xmpp-sidecar.service loaded activating auto-restart Carbonio Message Dispatcher XMPP sidecar pr> carbonio-message-dispatcher.service loaded activating auto-restart Carbonio Message Dispatcher daemon
carbonio-mta-sidecar.service loaded activating auto-restart Carbonio mta sidecar proxy
carbonio-preview-sidecar.service loaded activating auto-restart Carbonio Preview sidecar proxy
carbonio-preview.service loaded active running Carbonio Preview daemon
carbonio-prometheus-mysqld-exporter.service loaded active running Carbonio Prometheus MySQL exporter
carbonio-prometheus-nginx-exporter.service loaded active running Carbonio Prometheus NGINX exporter
carbonio-prometheus-node-exporter.service loaded active running Carbonio Prometheus Node Exporter
carbonio-prometheus-openldap-exporter.service loaded active running Carbonio Prometheus OpenLDAP exporter
carbonio-proxy-sidecar.service loaded activating auto-restart NGINX carbonio proxy sidecar
carbonio-storages-sidecar.service loaded activating auto-restart Storages-CE sidecar proxy
carbonio-storages.service loaded active running Carbonio Storages-CE daemon
carbonio-tasks-db-sidecar.service loaded activating auto-restart Carbonio Tasks database sidecar
carbonio-tasks-sidecar.service loaded activating auto-restart Carbonio Tasks sidecar proxy
carbonio-tasks.service loaded activating auto-restart Carbonio Tasks daemon
carbonio-user-management-sidecar.service loaded activating auto-restart Carbonio User Management sidecar proxy
carbonio-user-management.service loaded activating auto-restart Carbonio User Management daemon
carbonio-videoserver-sidecar.service loaded activating auto-restart Carbonio Videoserver sidecar proxy
carbonio-videoserver.service loaded active running Zextras Videoserver
carbonio-ws-collaboration-db-sidecar.service loaded activating auto-restart Carbonio Workstream Collaboration database > carbonio-ws-collaboration-sidecar.service loaded activating auto-restart Carbonio Workstream Collaboration sidecar p> carbonio-ws-collaboration.service loaded activating auto-restart Carbonio Workstream Collaboration daemon
carbonio.service loaded active exited LSB: Carbonio Services
3.
curl -s -v http://127.78.0.4:10000/health | jq
* Trying 127.78.0.4:10000...
* connect to 127.78.0.4 port 10000 failed: Connection refused
* Failed to connect to 127.78.0.4 port 10000 after 0 ms: Connection refused
* Closing connection 0
thanks in advance.. and thx for the troubleshooting link - i took a look at that before, but wasn't able to see anything relevant.... might need to read more...
Hello jajaish,
You can either support restarting the discovery service right then and there, or run the screen with the command below to see what errors occurred and resend the error log so people can support you more easily.
journalctl -u service-discover -f
it won't restart 🙁
jounalcontrol looks like this
Mar 04 05:24:35 zimbra systemd[1]: service-discover.service: Failed with result 'exit-code'. Mar 04 05:24:35 zimbra systemd[1]: Failed to start Service discover agent for Zextras. Mar 04 05:24:50 zimbra systemd[1]: service-discover.service: Scheduled restart job, restart counter is at 1810. Mar 04 05:24:50 zimbra systemd[1]: Stopped Service discover agent for Zextras. Mar 04 05:24:50 zimbra systemd[1]: Starting Service discover agent for Zextras... Mar 04 05:24:50 zimbra service-discoverd[690384]: 'grpc_tls' port already exists Mar 04 05:24:50 zimbra service-discoverd[690384]: local service-discover agent must NOT be present in ldap/zimbraServiceEnabled attribute zimbra.jaja.com Mar 04 05:24:50 zimbra systemd[1]: service-discover.service: Main process exited, code=exited, status=236/SMACK_PROCESS_LABEL
also let me add this one...
sudo systemctl status service-discover.service
● service-discover.service - Service discover agent for Zextras
Loaded: loaded (/lib/systemd/system/service-discover.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2025-03-04 05:33:02 PST; 9s ago
Docs: https://docs.zextras.com/
Process: 703351 ExecStart=/usr/bin/service-discoverd agent (code=exited, status=236/SMACK_PROCESS_LABEL)
Main PID: 703351 (code=exited, status=236/SMACK_PROCESS_LABEL)
CPU: 6ms
and this
ps faux |grep ldap carboni+ 990 0.0 0.0 1904024 6620 ? Ssl Mar03 0:03 /usr/bin/carbonio-prometheus-openldap-exporter --config /etc/carbonio/carbonio-prometheus-openldap-exporter/carbonio-prometheus-openldap-exporter.yml useruser+ 710676 0.0 0.0 6560 2148 pts/1 S+ 05:38 0:00 \_ grep --color=auto ldap zextras 1627 0.2 2.2 85117056 358272 ? Ssl Mar03 0:59 /opt/zextras/common/libexec/slapd -l LOCAL0 -h ldap://zimbra.jaja.com:389 ldapi:/// -F /opt/zextras/data/ldap/config zextras 5173 0.0 0.0 19296 10992 ? S Mar03 0:11 /usr/bin/perl -w /opt/zextras/libexec/zmstat-ldap
and i've tried the below commands as well...
carbonio ms `zmhostname` +zimbraServiceInstalled 'service-discover' +zimbraServiceEnabled 'service-discover' zmprov -l ms `zmhostname` +zimbraServiceInstalled 'service-discover' +zimbraServiceEnabled 'service-discover'
/opt/zextras/log/mailbox.log has alot of these...
org.apache.http.conn.HttpHostConnectException: Connect to localhost:8500 [localhost/127.0.0.1] failed: Connection refused
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MinimalClientExec.execute(MinimalClientExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.MinimalHttpClient.doExecute(MinimalHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at com.zextras.mailbox.client.ServiceDiscoverHttpClient.getConfig(ServiceDiscoverHttpClient.java:50) ~[mailbox.jar:?]
at com.zextras.mailbox.messageBroker.MessageBrokerFactory.getMessageBrokerClientInstance(MessageBrokerFactory.java:38) ~[mailbox.jar:?]
at io.vavr.control.Try.of(Try.java:75) ~[vavr-0.10.4.jar:?]
at com.zimbra.cs.service.admin.AdminService.getMessageBroker(AdminService.java:411) ~[mailbox.jar:?]
at com.zimbra.cs.service.admin.AdminService.lambda$scheduleRetry$2(AdminService.java:423) ~[mailbox.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:579) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:568) ~[?:?]
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
at java.net.Socket.connect(Socket.java:633) ~[?:?]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
... 16 more
2025-03-04 05:42:07,788 ERROR [pool-1922-thread-1] [] ServiceDiscoverHttpClient - Exception trying to get config from service discover:
org.apache.http.conn.HttpHostConnectException: Connect to localhost:8500 [localhost/127.0.0.1] failed: Connection refused
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MinimalClientExec.execute(MinimalClientExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.MinimalHttpClient.doExecute(MinimalHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at com.zextras.mailbox.client.ServiceDiscoverHttpClient.getConfig(ServiceDiscoverHttpClient.java:50) ~[mailbox.jar:?]
at com.zextras.mailbox.messageBroker.MessageBrokerFactory.getMessageBrokerClientInstance(MessageBrokerFactory.java:40) ~[mailbox.jar:?]
at io.vavr.control.Try.of(Try.java:75) ~[vavr-0.10.4.jar:?]
at com.zimbra.cs.service.admin.AdminService.getMessageBroker(AdminService.java:411) ~[mailbox.jar:?]
at com.zimbra.cs.service.admin.AdminService.lambda$scheduleRetry$2(AdminService.java:423) ~[mailbox.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:579) ~[?:?]
at sun.nio.ch.Net.connect(Net.java:568) ~[?:?]
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
at java.net.Socket.connect(Socket.java:633) ~[?:?]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
... 16 more
/opt/zextras/log/zmmailboxd.out
this looks ok.... (i think)
[10922.592s][info][gc] GC(12) Pause Young (Normal) (G1 Evacuation Pause) 1405M->94M(2970M) 15.465ms [12814.227s][info][gc] GC(13) Pause Young (Concurrent Start) (Metadata GC Threshold) 680M->110M(2970M) 26.316ms [12814.227s][info][gc] GC(14) Concurrent Mark Cycle [12814.343s][info][gc] GC(14) Pause Remark 111M->107M(2970M) 28.673ms [12814.380s][info][gc] GC(14) Pause Cleanup 107M->107M(2970M) 0.189ms [12814.383s][info][gc] GC(14) Concurrent Mark Cycle 156.854ms [15480.941s][info][gc] GC(15) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 1526M->141M(2970M) 26.861ms [15502.016s][info][gc] GC(16) Pause Young (Mixed) (G1 Evacuation Pause) 513M->123M(2970M) 26.293ms [15562.077s][info][gc] GC(17) Pause Young (Normal) (G1 Evacuation Pause) 1441M->151M(2970M) 17.109ms [15610.132s][info][gc] GC(18) Pause Young (Normal) (G1 Evacuation Pause) 1441M->146M(2970M) 15.377ms [15664.566s][info][gc] GC(19) Pause Young (Normal) (G1 Evacuation Pause) 1444M->149M(2970M) 15.305ms [15761.941s][info][gc] GC(20) Pause Young (Normal) (G1 Evacuation Pause) 1443M->160M(2970M) 20.542ms [15946.772s][info][gc] GC(21) Pause Young (Normal) (G1 Evacuation Pause) 1460M->172M(2970M) 22.819ms [16127.736s][info][gc] GC(22) Pause Young (Normal) (G1 Evacuation Pause) 1444M->177M(2970M) 24.695ms [16319.252s][info][gc] GC(23) Pause Young (Normal) (G1 Evacuation Pause) 1549M->171M(2970M) 22.462ms [16513.395s][info][gc] GC(24) Pause Young (Normal) (G1 Evacuation Pause) 1445M->177M(2970M) 25.553ms [16616.162s][info][gc] GC(25) Pause Young (Normal) (G1 Evacuation Pause) 1445M->177M(2970M) 24.615ms [16726.780s][info][gc] GC(26) Pause Young (Normal) (G1 Evacuation Pause) 1445M->178M(2970M) 30.596ms [16846.311s][info][gc] GC(27) Pause Young (Normal) (G1 Evacuation Pause) 1446M->182M(2970M) 25.834ms [16936.899s][info][gc] GC(28) Pause Young (Normal) (G1 Evacuation Pause) 1448M->184M(2970M) 98.525ms [17063.975s][info][gc] GC(29) Pause Young (Normal) (G1 Evacuation Pause) 1448M->184M(2970M) 26.728ms [17397.490s][info][gc] GC(30) Pause Young (Normal) (GCLocker Initiated GC) 1448M->179M(2970M) 27.970ms [18197.472s][info][gc] GC(31) Pause Young (Normal) (G1 Evacuation Pause) 1529M->183M(2970M) 35.196ms [19384.812s][info][gc] GC(32) Pause Young (Normal) (G1 Evacuation Pause) 1463M->183M(2970M) 32.876ms [20021.091s][info][gc] GC(33) Pause Young (Normal) (G1 Evacuation Pause) 1555M->193M(2970M) 28.188ms [20811.409s][info][gc] GC(34) Pause Young (Normal) (G1 Evacuation Pause) 1459M->192M(2970M) 31.937ms [21543.206s][info][gc] GC(35) Pause Young (Normal) (GCLocker Initiated GC) 1456M->196M(2970M) 27.366ms [22226.079s][info][gc] GC(36) Pause Young (Normal) (G1 Evacuation Pause) 1460M->192M(2970M) 26.773ms [23119.097s][info][gc] GC(37) Pause Young (Normal) (G1 Evacuation Pause) 1478M->204M(2970M) 33.180ms [23842.449s][info][gc] GC(38) Pause Young (Normal) (GCLocker Initiated GC) 1464M->215M(2970M) 29.625ms [24461.775s][info][gc] GC(39) Pause Young (Normal) (GCLocker Initiated GC) 1557M->211M(2970M) 28.308ms [25163.299s][info][gc] GC(40) Pause Young (Normal) (G1 Evacuation Pause) 1647M->210M(2970M) 53.722ms [25963.634s][info][gc] GC(41) Pause Young (Normal) (G1 Evacuation Pause) 1836M->228M(2970M) 60.429ms [26666.094s][info][gc] GC(42) Pause Young (Normal) (G1 Evacuation Pause) 1506M->212M(2970M) 26.330ms [27663.561s][info][gc] GC(43) Pause Young (Normal) (G1 Evacuation Pause) 1626M->194M(2970M) 27.845ms
Could you please show me the output of the following command?
dpkg -l | grep service-discover
I think the problem lies with the service-discover service not running, and I believe reading the journal logs would provide some clues about the error.
If fixing service-discover is not effective, you might consider deleting the MESH data and reconfiguring it. Before deleting the MESH data, I think you should have a backup plan.
systemctl stop carbonio-service-discover rm -rf /var/lib/service-discover/data/ systemctl start carbonio-service-discover
After deleting, you can reconfigure it as published on the Carbonio homepage:
https://docs.zextras.com/carbonio-ce/html/admincli/mesh/credentials.html#mesh-credentials
Sorry, I misspelled the service name in the MESH data deletion command.
systemctl stop service-discover rm -rf /var/lib/service-discover/data/ systemctl start service-discover
thank you in advance
dpkg -l | grep service-discover rc service-discover-agent 0.2.1-1ubuntu amd64 Service discover agent cli ii service-discover-base 1.19.1-1jammy amd64 Service discover binary, based on HashiCorp Consul ii service-discover-daemon 0.2.1-1ubuntu amd64 Service discover daemon ii service-discover-server 0.2.1-1ubuntu amd64 Service discover server cli ii service-discover-template 0.36.0-1ubuntu amd64 Template rendering, notifier, and supervisor for Carbonioservice-discover
Oh, I saw you installed both service-discover-server and service-discover-agent. The conflict might be that service-discovery is not starting the service. Please remove the service-discover-agent installation package in the following order and restart the service to see service-discover:
# apt purge service-discover-agent # apt clean # apt autoremove
Please check if the service-discover service is running?
# systemctl restart service-discover
Thanks @darkma91cian for the observation. @jajaish Yes, we do not need to install service-discover-agent in the single server scenario, and specially when service-discover-server is installed on the same node. Installing both of them in the same node would create conflict.
Regards,