Strange errors today. Updated 2 weeks ago from 24.5 to 24.7.1. But now some users cant log in. They get error 502 and 504. Other users works without problems. For the record I am not using "files" or "tasks". I am so worried that something is bad, but after the reboot (at the end of this message) everything look fine. No more error 502 or 504. But the command "systemctl status carbonio*" does not work anymore.
Stopped the server (when having problem) with the following result: zextras@master:~$ zmcontrol stop
Host master.sigtrap.org
Stopping config service...Done.
Stopping service webapp...Done.
Stopping stats...Done.
Stopping mta...Failed.
postfix/postlog: stopping the Postfix mail system
Stopping saslauthd...done.
Stopping milter server...failed.
Stopping cbpolicyd...Done.
Stopping opendkim...Done.
Stopping amavis...Done.
Stopping antivirus...Done.
Stopping antispam...Done.
Stopping proxy...Done.
Stopping memcached...Done.
Stopping mailbox...Done.
Stopping directory-server...Done.
AFTER REBOOT, looks like no more error 502 or 504. Maybe just a memory problem?
After reboot:
zextras@master:~$ zmcontrol -v
Carbonio Release 24.7.1
zextras@master:~$ zmcontrol status
Host master.sigtrap.org
amavis Running
antispam Running
antivirus Running
directory-server Running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
service-discover Running
stats Running
config service Running
root@master:/var/log# consul members
Node Address Status Type Build Protocol DC Segment
server-master-sigtrap-org 192.168.0.3:8301 alive server 1.10.12 2 dc1 <all>
root@master:/var/log# systemctl list-units carbonio*
UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
root@master:/var/log# journalctl -u carbonio-user-management.service
-- Logs begin at Sun 2024-09-08 12:57:54 CEST, end at Sun 2024-09-15 10:37:38 CEST. --
Sep 15 09:53:57 master.sigtrap.org systemd[1]: Stopping Carbonio User Management daemon...
Sep 15 09:53:57 master.sigtrap.org systemd[1]: carbonio-user-management.service: Main process exited, code=exited, status=143/n/a
Sep 15 09:53:57 master.sigtrap.org systemd[1]: carbonio-user-management.service: Failed with result 'exit-code'.
Sep 15 09:53:57 master.sigtrap.org systemd[1]: Stopped Carbonio User Management daemon.
-- Reboot --
Sep 15 09:55:46 master.sigtrap.org systemd[1]: Started Carbonio User Management daemon.
Sep 15 09:56:02 master.sigtrap.org java[1066]: Exception in thread "main" com.google.inject.ProvisionException: Unable to provision, see the following errors:
Sep 15 09:56:02 master.sigtrap.org java[1066]: 1) [Guice/ErrorInCustomProvider]: InaccessibleWSDLException: 2 counts of InaccessibleWSDLException.
Sep 15 09:56:02 master.sigtrap.org java[1066]: ConnectException: Connection refused
Sep 15 09:56:02 master.sigtrap.org java[1066]: ConnectException: Connection refused
Sep 15 09:56:02 master.sigtrap.org java[1066]: at UserManagementModule.provideServiceClient(UserManagementModule.java:54)
Sep 15 09:56:02 master.sigtrap.org java[1066]: while locating ServiceClient
Sep 15 09:56:02 master.sigtrap.org java[1066]: Learn more:
Sep 15 09:56:02 master.sigtrap.org java[1066]: https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER
Sep 15 09:56:02 master.sigtrap.org java[1066]: 1 error
Sep 15 09:56:02 master.sigtrap.org java[1066]: ======================
Sep 15 09:56:02 master.sigtrap.org java[1066]: Full classname legend:
Sep 15 09:56:02 master.sigtrap.org java[1066]: ======================
Sep 15 09:56:02 master.sigtrap.org java[1066]: ConnectException: "java.net.ConnectException"
Sep 15 09:56:02 master.sigtrap.org java[1066]: InaccessibleWSDLException: "com.sun.xml.ws.wsdl.parser.InaccessibleWSDLException"
Sep 15 09:56:02 master.sigtrap.org java[1066]: ServiceClient: "com.zextras.mailbox.client.service.ServiceClient"
Sep 15 09:56:02 master.sigtrap.org java[1066]: UserManagementModule: "com.zextras.carbonio.user_management.config.UserManagementModule"
Sep 15 09:56:02 master.sigtrap.org java[1066]: ========================
Sep 15 09:56:02 master.sigtrap.org java[1066]: End of classname legend:
Sep 15 09:56:02 master.sigtrap.org java[1066]: ========================
Sep 15 09:56:02 master.sigtrap.org java[1066]: at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
Sep 15 09:56:02 master.sigtrap.org java[1066]: at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1151)
Sep 15 09:56:02 master.sigtrap.org java[1066]: at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1186)
Sep 15 09:56:02 master.sigtrap.org java[1066]: at com.zextras.carbonio.user_management.Boot.main(Boot.java:31)
Sep 15 09:56:02 master.sigtrap.org java[1066]: Caused by: com.sun.xml.ws.wsdl.parser.InaccessibleWSDLException: 2 counts of InaccessibleWSDLException.
Sep 15 09:56:02 master.sigtrap.org java[1066]: java.net.ConnectException: Connection refused
Sep 15 09:56:02 master.sigtrap.org java[1066]: java.net.ConnectException: Connection refused
Sep 15 09:56:02 master.sigtrap.org java[1066]: at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.tryWithMex(RuntimeWSDLParser.java:242)
At boot:
[ 5.094467] systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[ 5.114256] systemd[1]: Configuration file /etc/systemd/system/carbonio-user-management.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[ 5.118467] systemd[1]: Configuration file /etc/systemd/system/carbonio-tasks.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[ 5.120769] systemd[1]: Configuration file /lib/systemd/system/carbonio-storages.service is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 5.122732] systemd[1]: Configuration file /etc/systemd/system/carbonio-storages.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[ 5.127091] systemd[1]: Configuration file /lib/systemd/system/carbonio-storages-sidecar.service is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 5.130955] systemd[1]: Configuration file /etc/systemd/system/carbonio-preview.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[ 5.136707] systemd[1]: Configuration file /etc/systemd/system/carbonio-files.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[ 5.139381] systemd[1]: Configuration file /etc/systemd/system/carbonio-docs-editor.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[ 5.144850] systemd[1]: Configuration file /etc/systemd/system/carbonio-docs-connector.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
When starting server from carbonio.log:
Sep 15 09:58:03 master amavis[5942]: (05942-01) ESMTP :10024 /opt/zextras/data/amavisd/tmp/amavis-20240915T095803-05942-cVa0en5y: <zextras@sigtrap.org> -> <zextras@sigtrap.org> SIZE=698 BODY=8BITMIME Received: from master.sigtrap.org ([127.0.0.1]) by localhost (master.sigtrap.org [127.0.0.1]) (amavis, port 10024) with ESMTP for <zextras@sigtrap.org>; Sun, 15 Sep 2024 09:58:03 +0200 (CEST)
Sep 15 09:58:03 master amavis[5942]: (05942-01) Checking: DxOy1vK3HKoI [127.0.0.1] <zextras@sigtrap.org> -> <zextras@sigtrap.org>
Sep 15 09:58:03 master amavis[5942]: (05942-01) (!)clamav: while feeding req_id 2: Error writing 142 bytes to socket: Broken pipe at /opt/zextras/common/lib/perl5/Amavis/IO/RW.pm line 325.
Sep 15 09:58:03 master amavis[5942]: (05942-01) (!)clamav: aborting: 2 pending, 0 remaining
Sep 15 09:58:03 master amavis[5942]: (05942-01) (!)ClamAV-remote-stream av-scanner FAILED: run_av error: clamav: ABORTED: Error writing 142 bytes to socket: Broken pipe at /opt/zextras/common/lib/perl5/Amavis/IO/RW.pm line 325. at /opt/zextras/common/lib/perl5/Amavis/AV.pm line 520.\n
Sep 15 09:58:03 master amavis[5942]: (05942-01) (!)WARN: all primary virus scanners failed, considering backups
Sep 15 09:58:03 master amavis[5942]: (05942-01) (!)connect to /run/carbonio/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /run/carbonio/clamd.socket: No such file or directory
Sep 15 09:58:03 master amavis[5942]: (05942-01) ClamAV-clamd: All attempts (1) failed connecting to /run/carbonio/clamd.socket, retrying (1)
Sep 15 09:58:03 master amavis[5944]: (05944-01) ESMTP :10024 /opt/zextras/data/amavisd/tmp/amavis-20240915T095802-05944-oBI1tREZ: <zextras@sigtrap.org> -> <zextras@sigtrap.org> SIZE=698 BODY=8BITMIME Received: from master.sigtrap.org ([127.0.0.1]) by localhost (master.sigtrap.org [127.0.0.1]) (amavis, port 10024) with ESMTP for <zextras@sigtrap.org>; Sun, 15 Sep 2024 09:58:02 +0200 (CEST)
Sep 15 09:58:03 master amavis[5944]: (05944-01) Checking: Wrm8YHH6y-ih [127.0.0.1] <zextras@sigtrap.org> -> <zextras@sigtrap.org>
Sep 15 09:58:03 master amavis[5944]: (05944-01) (!)clamav: while feeding req_id 2: Error writing 142 bytes to socket: Broken pipe at /opt/zextras/common/lib/perl5/Amavis/IO/RW.pm line 325.
Sep 15 09:58:03 master amavis[5944]: (05944-01) (!)clamav: aborting: 2 pending, 0 remaining
Sep 15 09:58:03 master amavis[5944]: (05944-01) (!)ClamAV-remote-stream av-scanner FAILED: run_av error: clamav: ABORTED: Error writing 142 bytes to socket: Broken pipe at /opt/zextras/common/lib/perl5/Amavis/IO/RW.pm line 325. at /opt/zextras/common/lib/perl5/Amavis/AV.pm line 520.\n
Sep 15 09:58:03 master amavis[5944]: (05944-01) (!)WARN: all primary virus scanners failed, considering backups
Sep 15 09:58:03 master amavis[5944]: (05944-01) (!)connect to /run/carbonio/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /run/carbonio/clamd.socket: No such file or directory
Sep 15 09:58:03 master amavis[5944]: (05944-01) ClamAV-clamd: All attempts (1) failed connecting to /run/carbonio/clamd.socket, retrying (1)
Sep 15 09:58:04 master amavis[5942]: (05942-01) (!)connect to /run/carbonio/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /run/carbonio/clamd.socket: No such file or directory
Sep 15 09:58:04 master amavis[5942]: (05942-01) (!)ClamAV-clamd: All attempts (1) failed connecting to /run/carbonio/clamd.socket, retrying (2)
Sep 15 09:58:04 master amavis[5944]: (05944-01) (!)connect to /run/carbonio/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /run/carbonio/clamd.socket: No such file or directory
Sep 15 09:58:04 master amavis[5944]: (05944-01) (!)ClamAV-clamd: All attempts (1) failed connecting to /run/carbonio/clamd.socket, retrying (2)
Sep 15 09:58:10 master amavis[5942]: (05942-01) (!)connect to /run/carbonio/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /run/carbonio/clamd.socket: No such file or directory
Sep 15 09:58:10 master amavis[5942]: (05942-01) (!)ClamAV-clamd av-scanner FAILED: run_av error: Too many retries to talk to /run/carbonio/clamd.socket (All attempts (1) failed connecting to /run/carbonio/clamd.socket) at /opt/zextras/common/lib/perl5/Amavis/AV.pm line 663.\n
Sep 15 09:58:10 master amavis[5942]: (05942-01) (!!)AV: ALL VIRUS SCANNERS FAILED
Sep 15 09:58:10 master amavis[5944]: (05944-01) (!)connect to /run/carbonio/clamd.socket failed, attempt #1: Can't connect to a UNIX socket /run/carbonio/clamd.socket: No such file or directory
Sep 15 09:58:10 master amavis[5944]: (05944-01) (!)ClamAV-clamd av-scanner FAILED: run_av error: Too many retries to talk to /run/carbonio/clamd.socket (All attempts (1) failed connecting to /run/carbonio/clamd.socket) at /opt/zextras/common/lib/perl5/Amavis/AV.pm line 663.\n
Sep 15 09:58:10 master amavis[5944]: (05944-01) (!!)AV: ALL VIRUS SCANNERS FAILED
From syslog when booting:
Sep 15 09:55:46 master service-discoverd[1080]: panic: runtime error: invalid memory address or nil pointer dereference
Sep 15 09:55:46 master service-discoverd[1080]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x5e908f]
Sep 15 09:55:46 master service-discoverd[1080]: goroutine 1 [running]:
Sep 15 09:55:46 master service-discoverd[1080]: github.com/go-ldap/ldap/v3.(*Conn).nextMessageID(...)
Sep 15 09:55:46 master service-discoverd[1080]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/conn.go:282
Sep 15 09:55:46 master service-discoverd[1080]: github.com/go-ldap/ldap/v3.(*Conn).doRequest(0x0, {0x6c8140, 0xc000125c00})
Sep 15 09:55:46 master service-discoverd[1080]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/request.go:26 +0x4f
Sep 15 09:55:46 master service-discoverd[1080]: github.com/go-ldap/ldap/v3.(*Conn).SimpleBind(0x0, 0x1?)
Sep 15 09:55:46 master service-discoverd[1080]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/bind.go:65 +0x105
Sep 15 09:55:46 master service-discoverd[1080]: github.com/go-ldap/ldap/v3.(*Conn).Bind(0x0, {0xc0001147a0, 0x1e}, {0xc0000b0a36, 0xa})
Sep 15 09:55:46 master service-discoverd[1080]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/bind.go:104 +0x8c
Sep 15 09:55:46 master service-discoverd[1080]: github.com/Zextras/service-discover/cli/lib/carbonio.connect(0xc000096540, 0x0?)
Sep 15 09:55:46 master service-discoverd[1080]: #011/tmp/cli/lib/carbonio/ldap.go:222 +0x1b3
Sep 15 09:55:46 master service-discoverd[1080]: github.com/Zextras/service-discover/cli/lib/carbonio.(*ldapContext).QueryAllServersWithService(0xc0000a8a20?, {0x65a66f, 0x10})
Sep 15 09:55:46 master service-discoverd[1080]: #011/tmp/cli/lib/carbonio/ldap.go:159 +0x65
Sep 15 09:55:46 master service-discoverd[1080]: main.queryAllServiceDiscoverServers({0x6c9d10?, 0xc000096540?})
Sep 15 09:55:46 master service-discoverd[1080]: #011/tmp/service-discoverd/service-discoverd.go:281 +0x31
Sep 15 09:55:46 master service-discoverd[1080]: main.runServiceDiscoverDaemon({0x6caca0, 0x895880}, {0xc00008e020?, 0x805968?, 0xc0000061c0?})
Sep 15 09:55:46 master service-discoverd[1080]: #011/tmp/service-discoverd/service-discoverd.go:115 +0x3ba
Sep 15 09:55:46 master service-discoverd[1080]: main.main()
Sep 15 09:55:46 master service-discoverd[1080]: #011/tmp/service-discoverd/service-discoverd.go:83 +0x36
Sep 15 09:55:46 master systemd[1]: service-discover.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 15 09:55:46 master systemd[1]: service-discover.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master systemd[1]: Failed to start Service discover server for Zextras.
Sep 15 09:55:46 master consul[1021]: Failed looking up sidecar proxy info for carbonio-docs-editor: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:55:46 master consul[1065]: Failed looking up sidecar proxy info for carbonio-user-management: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:55:46 master consul[1061]: Failed looking up sidecar proxy info for carbonio-tasks: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:55:46 master systemd[1]: carbonio-docs-editor-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 09:55:46 master systemd[1]: carbonio-docs-editor-sidecar.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master systemd[1]: carbonio-user-management-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 09:55:46 master systemd[1]: carbonio-user-management-sidecar.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master systemd[1]: carbonio-tasks-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 09:55:46 master systemd[1]: carbonio-tasks-sidecar.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master consul[1031]: Failed looking up sidecar proxy info for carbonio-mailbox-admin: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:55:46 master systemd[1]: carbonio-mailbox-admin-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 09:55:46 master systemd[1]: carbonio-mailbox-admin-sidecar.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master consul[1035]: Failed looking up sidecar proxy info for carbonio-preview: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:55:46 master systemd[1]: carbonio-preview-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 09:55:46 master systemd[1]: carbonio-preview-sidecar.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master consul[1051]: Failed looking up sidecar proxy info for carbonio-proxy: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:55:46 master consul[1015]: Failed looking up sidecar proxy info for carbonio-docs-connector: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:55:46 master systemd[1]: carbonio-proxy-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 09:55:46 master systemd[1]: carbonio-proxy-sidecar.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master systemd[1]: carbonio-docs-connector-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 09:55:46 master systemd[1]: carbonio-docs-connector-sidecar.service: Failed with result 'exit-code'.
Sep 15 09:55:46 master consul[1053]: Failed looking up sidecar proxy info for carbonio-storages: Get "http://127.0.0.1:8500/v1/agent/services": dial tcp 127.0.0.1:8500: connect: connection refused
Sep 15 09:56:01 master systemd[1]: service-discover.service: Scheduled restart job, restart counter is at 1.
Sep 15 09:56:01 master systemd[1]: Stopped Service discover server for Zextras.
Sep 15 09:56:01 master systemd[1]: Starting Service discover server for Zextras...
Sep 15 09:56:01 master service-discoverd[1809]: panic: runtime error: invalid memory address or nil pointer dereference
Sep 15 09:56:01 master service-discoverd[1809]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x5e908f]
Sep 15 09:56:01 master service-discoverd[1809]: goroutine 1 [running]:
Sep 15 09:56:01 master service-discoverd[1809]: github.com/go-ldap/ldap/v3.(*Conn).nextMessageID(...)
Sep 15 09:56:01 master service-discoverd[1809]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/conn.go:282
Sep 15 09:56:01 master service-discoverd[1809]: github.com/go-ldap/ldap/v3.(*Conn).doRequest(0x0, {0x6c8140, 0xc00017dc40})
Sep 15 09:56:01 master service-discoverd[1809]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/request.go:26 +0x4f
Sep 15 09:56:01 master service-discoverd[1809]: github.com/go-ldap/ldap/v3.(*Conn).SimpleBind(0x0, 0x1?)
Sep 15 09:56:01 master service-discoverd[1809]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/bind.go:65 +0x105
Sep 15 09:56:01 master service-discoverd[1809]: github.com/go-ldap/ldap/v3.(*Conn).Bind(0x0, {0xc0001707a0, 0x1e}, {0xc000014a66, 0xa})
Sep 15 09:56:01 master service-discoverd[1809]: #011/root/go/pkg/mod/github.com/go-ldap/ldap/v3@v3.2.4/bind.go:104 +0x8c
Sep 15 09:56:01 master service-discoverd[1809]: github.com/Zextras/service-discover/cli/lib/carbonio.connect(0xc000076540, 0x0?)
Sep 15 09:56:01 master service-discoverd[1809]: #011/tmp/cli/lib/carbonio/ldap.go:222 +0x1b3
Sep 15 09:56:01 master service-discoverd[1809]: github.com/Zextras/service-discover/cli/lib/carbonio.(*ldapContext).QueryAllServersWithService(0xc00010ca20?, {0x65a66f, 0x10})
Sep 15 09:56:01 master service-discoverd[1809]: #011/tmp/cli/lib/carbonio/ldap.go:159 +0x65
Sep 15 09:56:01 master service-discoverd[1809]: main.queryAllServiceDiscoverServers({0x6c9d10?, 0xc000076540?})
Sep 15 09:56:01 master service-discoverd[1809]: #011/tmp/service-discoverd/service-discoverd.go:281 +0x31
Sep 15 09:56:01 master service-discoverd[1809]: main.runServiceDiscoverDaemon({0x6caca0, 0x895880}, {0xc000016040?, 0x805968?, 0xc0000061c0?})
Sep 15 09:56:01 master service-discoverd[1809]: #011/tmp/service-discoverd/service-discoverd.go:115 +0x3ba
Sep 15 09:56:01 master service-discoverd[1809]: main.main()
Sep 15 09:56:01 master service-discoverd[1809]: #011/tmp/service-discoverd/service-discoverd.go:83 +0x36
Sep 15 09:56:01 master systemd[1]: service-discover.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 15 09:56:01 master systemd[1]: service-discover.service: Failed with result 'exit-code'.
Sep 15 09:56:01 master systemd[1]: Failed to start Service discover server for Zextras.
Sep 15 09:56:02 master java[1020]: 09:56:02.660 [main] INFO o.jboss.resteasy.plugins.guice.i18n - RESTEASY011025: registering provider instance for com.zextras.carbonio.docs_connector.auth.AccessTokenValidationFilter
Sep 15 09:56:02 master java[1020]: 09:56:02.683 [main] INFO o.jboss.resteasy.plugins.guice.i18n - RESTEASY011025: registering provider instance for com.zextras.carbonio.docs_connector.auth.CookieAuthenticationFilter
Sep 15 09:56:02 master java[1020]: 09:56:02.684 [main] INFO o.jboss.resteasy.plugins.guice.i18n - RESTEASY011020: registering factory for com.zextras.carbonio.docs_connector.controllers.HealthController
Sep 15 09:56:02 master java[1020]: 09:56:02.801 [main] INFO o.jboss.resteasy.plugins.guice.i18n - RESTEASY011020: registering factory for com.zextras.carbonio.docs_connector.controllers.FilesController
Sep 15 09:56:02 master java[1020]: 09:56:02.820 [main] INFO o.jboss.resteasy.plugins.guice.i18n - RESTEASY011020: registering factory for com.zextras.carbonio.docs_connector.controllers.WopiController
Sep 15 09:56:03 master carbonio-tasks[1128]: 09:56:03.030 [main] WARN c.z.c.t.c.ServiceDiscoverHttpClient - Unable to decode the ServiceDiscover response when 'db-username' key config is retrieved. org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8500 [localhost/127.0.0.1] failed: Connection refused
Sep 15 09:56:03 master carbonio-tasks[1128]: 09:56:03.039 [main] WARN c.z.c.t.c.ServiceDiscoverHttpClient - Unable to decode the ServiceDiscover response when 'db-password' key config is retrieved. org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8500 [localhost/127.0.0.1] failed: Connection refused
Sep 15 09:56:03 master carbonio-tasks[1128]: 09:56:03.042 [main] WARN c.z.c.t.c.ServiceDiscoverHttpClient - Unable to decode the ServiceDiscover response when 'db-name' key config is retrieved. org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8500 [localhost/127.0.0.1] failed: Connection refused
Sep 15 09:56:03 master carbonio-tasks[1128]: 09:56:03.045 [main] WARN c.z.c.t.c.ServiceDiscoverHttpClient - Unable to decode the ServiceDiscover response when 'hikari-max-pool-size' key config is retrieved. org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8500 [localhost/127.0.0.1] failed: Connection refused
Sep 15 09:56:03 master carbonio-tasks[1128]: 09:56:03.050 [main] WARN c.z.c.t.c.ServiceDiscoverHttpClient - Unable to decode the ServiceDiscover response when 'hikari-min-idle-connections' key config is retrieved. org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8500 [localhost/127.0.0.1] failed: Connection refused
Sep 15 09:56:04 master carbonio-files[1123]: 09:56:04.298 [main] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Exception during pool initialization.
Sep 15 09:56:04 master carbonio-files[1123]: org.postgresql.util.PSQLException: Connection to 127.78.0.2:20000 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Sep 15 10:23:21 master carbonio-preview-start.sh[1696]: "[2024-09-15 10:23:21,104] INFO [uvicorn.access.send:482] 127.0.0.1:36624 - "GET /health/ready/ HTTP/1.1" 200"
Sep 15 10:23:21 master service-discoverd[2403]: 2024-09-15T10:23:21.561+0200 [WARN] agent: Check is now critical: check=service:carbonio-tasks
Sep 15 10:23:21 master carbonio-tasks[23115]: 10:23:21.780 [main] ERROR ROOT - Service stopped unexpectedly:
Sep 15 10:23:21 master carbonio-tasks[23115]: org.flywaydb.core.internal.exception.FlywaySqlException: Unable to obtain connection from database: The server requested SCRAM-based authentication, but the password is an empty string.
Sep 15 10:23:21 master carbonio-tasks[23115]: --------------------------------------------------------------------------------------------------------------------------------
Sep 15 10:23:21 master carbonio-tasks[23115]: SQL State : 08004
Sep 15 10:23:21 master carbonio-tasks[23115]: Error Code : 0
Sep 15 10:23:21 master carbonio-tasks[23115]: Message : The server requested SCRAM-based authentication, but the password is an empty string.
Sep 15 10:23:21 master carbonio-tasks[23115]: #011at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:60)
Should I be worried?
I was in the wrong directory for systemctl list-units carbonio*
root@master:~# systemctl list-units carbonio*
UNIT LOAD ACTIVE SUB DESCRIPTION
carbonio-clamav-sidecar.service loaded active running Carbonio clamav sidecar proxy
carbonio-clamav-signature-provider-sidecar.service loaded active running NGINX Carbonio ClamAV signature provider sidecar
carbonio-docs-connector-sidecar.service loaded active running Carbonio docs connector sidecar proxy
carbonio-docs-connector.service loaded active running Carbonio docs connector daemon
carbonio-docs-editor-sidecar.service loaded active running Carbonio Docs server sidecar proxy
carbonio-docs-editor.service loaded active running Carbonio Docs Server
carbonio-files-db-sidecar.service loaded active running Carbonio Files database sidecar
carbonio-files-sidecar.service loaded active running Carbonio Files sidecar proxy
carbonio-files.service loaded active running Carbonio Files daemon
carbonio-mailbox-admin-sidecar.service loaded active running Carbonio Mailbox Admin proxy sidecar
carbonio-mailbox-nslookup-sidecar.service loaded active running Carbonio Mailbox NSLookup proxy sidecar
carbonio-mailbox-sidecar.service loaded active running Mailbox carbonio proxy sidecar
carbonio-mta-sidecar.service loaded active running Carbonio mta sidecar proxy
carbonio-preview-sidecar.service loaded active running Carbonio Preview sidecar proxy
carbonio-preview.service loaded active running Carbonio Preview daemon
carbonio-prometheus-consul-exporter.service loaded active running Carbonio Prometheus Consul exporter
carbonio-prometheus-mysqld-exporter.service loaded active running Carbonio Prometheus MySQL exporter
carbonio-prometheus-nginx-exporter.service loaded active running Carbonio Prometheus NGINX exporter
carbonio-prometheus-node-exporter.service loaded active running Carbonio Prometheus Node Exporter
carbonio-prometheus-openldap-exporter.service loaded active running Carbonio Prometheus OpenLDAP exporter
carbonio-proxy-sidecar.service loaded active running NGINX carbonio proxy sidecar
carbonio-storages-sidecar.service loaded active running Storages-CE sidecar proxy
carbonio-storages.service loaded active running Carbonio Storages-CE daemon
carbonio-tasks-db-sidecar.service loaded active running Carbonio Tasks database sidecar
carbonio-tasks-sidecar.service loaded active running Carbonio Tasks sidecar proxy
carbonio-tasks.service loaded activating auto-restart Carbonio Tasks daemon
carbonio-user-management-sidecar.service loaded active running Carbonio User Management sidecar proxy
carbonio-user-management.service loaded active running Carbonio User Management daemon
● carbonio.service loaded failed failed LSB: Carbonio Services
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
29 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
Try this: stop carbonio, empty /opt/zextras/jetty/work, start carbonio.
Well, forget the folder cleaning, I just had the same issue.
The swap was full and services eventually stopped.
I rebooted the server and right once rebooted, it's already full of zombie processes (24 zombies process less than 2 minutes after reboot).
ps aux shows this:
carboni+ 2399 0.7 0.6 2290532 51584 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400d14a2d2f-bootstrap.json --disable-hot-restart carboni+ 2400 0.8 0.6 2290532 52084 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400db409679-bootstrap.json --disable-hot-restart carboni+ 2401 0.9 0.6 2290532 50960 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400dcafecbc-bootstrap.json --disable-hot-restart carboni+ 2402 0.7 0.6 2290532 49952 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400e2d2d068-bootstrap.json --disable-hot-restart carboni+ 2403 0.8 0.6 2290532 51004 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400d7907f28-bootstrap.json --disable-hot-restart carboni+ 2430 1.3 0.6 2290532 54104 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400f3f69ada-bootstrap.json --disable-hot-restart carboni+ 2438 0.6 0.6 2290532 50996 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400e883be6b-bootstrap.json --disable-hot-restart carboni+ 2441 0.7 0.6 2290532 49896 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400df2a3037-bootstrap.json --disable-hot-restart carboni+ 2461 0.6 0.6 2290532 49680 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400e961bae8-bootstrap.json --disable-hot-restart carboni+ 2487 0.7 0.6 2290532 50296 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f664010ffd526b-bootstrap.json --disable-hot-restart carboni+ 2488 0.9 0.6 2290532 50904 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f664010de38089-bootstrap.json --disable-hot-restart carboni+ 2490 0.8 0.6 2290532 49980 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f664010b345ae7-bootstrap.json --disable-hot-restart carboni+ 2494 1.2 0.6 2290532 53884 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66401159a6c1b-bootstrap.json --disable-hot-restart carboni+ 2495 0.7 0.6 2290532 50132 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f6640117d6aff5-bootstrap.json --disable-hot-restart carboni+ 2496 0.6 0.6 2290532 49700 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f6640106739307-bootstrap.json --disable-hot-restart carboni+ 2498 0.8 0.6 2290532 50072 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f6640109d6ccad-bootstrap.json --disable-hot-restart carboni+ 2499 1.0 0.6 2290532 51356 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f6640100785237-bootstrap.json --disable-hot-restart carboni+ 2502 0.8 0.6 2290532 51100 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f664011af10929-bootstrap.json --disable-hot-restart carboni+ 2503 0.8 0.6 2290532 53236 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f664010225db6e-bootstrap.json --disable-hot-restart carboni+ 2505 0.7 0.6 2290532 49764 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f6640115537c28-bootstrap.json --disable-hot-restart carboni+ 2506 1.8 0.6 2290532 52192 ? Ssl 16:37 0:01 /usr/bin/envoy --config-path /tmp/envoy-17f664010bb6da9f-bootstrap.json --disable-hot-restart carboni+ 2564 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2565 1.9 0.6 2290532 52444 ? Ssl 16:37 0:01 /usr/bin/envoy --config-path /tmp/envoy-17f6640108ecfb09-bootstrap.json --disable-hot-restart carboni+ 2576 0.8 0.6 2290532 51140 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66401013e4323-bootstrap.json --disable-hot-restart videose+ 2584 0.8 0.6 2290532 51456 ? Ssl 16:37 0:00 /usr/bin/envoy --config-path /tmp/envoy-17f66400fc1b9bba-bootstrap.json --disable-hot-restart carboni+ 2623 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2645 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2660 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2665 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2677 0.2 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2689 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2695 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2713 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> videose+ 2731 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2746 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2749 0.2 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2755 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2760 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2782 0.2 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2789 0.2 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2797 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2798 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2818 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2834 0.2 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2849 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2850 0.2 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2856 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct> carboni+ 2872 0.1 0.0 0 0 ? Z 16:37 0:00 [consul] <defunct>
There was even more processes like that before the reboot.