After update to Ubuntu 22.04 and Carbonio 24.3.0 we faced such problems:
1. After logrotate (at 00:00), the system stops writing to the /var/log/carbonio.log file. The file occupies 0 bytes, but at the same time all entries in the journalctl go as usual. If we do not reboot the system, then at the end of the day an empty daily report comes. Also the log archive (.gz) is not created.
2. We see errors in the operation of services:
carbonio-message-dispatcher.service - Carbonio Message Dispatcher daemon
Loaded: loaded (/lib/systemd/system/carbonio-message-dispatcher.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/carbonio-message-dispatcher.service.d
└─override.conf
Active: active (running) since Wed 2024-03-27 11:03:24 EET; 1h 19min ago
Docs: https://docs.zextras.com/
Main PID: 4573 (mongooseimctl)
Tasks: 45 (limit: 38352)
Memory: 105.5M
CPU: 15.425s
CGroup: /system.slice/carbonio-message-dispatcher.service
├─4573 /bin/bash /opt/zextras/common/bin/mongooseimctl foreground
├─4583 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/beam.smp -Bd -C multi_time_warp -- -root /opt/zextras/common/lib/mongooseim -bindir /opt/zextras/common/lib/mongooseim/erts-14.1/bin ->
├─4619 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/epmd -daemon
├─4625 erl_child_setup 1024
├─5128 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/inet_gethost 4
└─5129 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/inet_gethost 4
мар 27 11:03:24 mail2.*** systemd[1]: Started Carbonio Message Dispatcher daemon.
мар 27 11:03:24 mail2. mongooseimctl[4583]: Root: /opt/zextras/common/lib/mongooseim
мар 27 11:03:24 mail2. mongooseimctl[4583]: Exec: /opt/zextras/common/lib/mongooseim/erts-14.1/bin/erlexec -noshell -noinput +Bd -boot /opt/zextras/common/lib/mongooseim/releases/6.2.0/start >
мар 27 11:03:25 mail2. mongooseimctl[4583]: 11:03:25.049 [error] Failed to open log file /opt/zextras/common/lib/mongooseim/log/console.log with error permission denied
мар 27 11:03:25 mail2. mongooseimctl[4583]: 11:03:25.049 [error] Failed to open log file /opt/zextras/common/lib/mongooseim/log/error.log with error permission denied
мар 27 11:03:25 mail2. mongooseimctl[4583]: 11:03:25.520 [error] Failed to open crash log file /opt/zextras/common/lib/mongooseim/log/crash.log with error: permission denied
мар 27 11:03:27 mail2. mongooseimctl[4583]: when=2024-03-27T09:03:27.502767+00:00 level=notice what=mongooseim_node_started pid=<0.293.0> at=ejabberd_app:do_start/0:83 version=6.2.0 node=mong>
carbonio-prometheus.service - Carbonio Prometheus service
Loaded: loaded (/lib/systemd/system/carbonio-prometheus.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-03-27 11:02:53 EET; 1h 20min ago
Main PID: 892 (prometheus)
Tasks: 18 (limit: 38352)
Memory: 1020.0M
CPU: 51.375s
CGroup: /system.slice/carbonio-prometheus.service
└─892 /usr/bin/prometheus --config.file=/etc/carbonio/carbonio-prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data --storage.tsdb.retention.time=180d --enable-feature=memo>
мар 27 12:20:27 mail2. prometheus[892]: ts=2024-03-27T10:20:27.941Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:20:42 mail2. prometheus[892]: ts=2024-03-27T10:20:42.943Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:20:57 mail2. prometheus[892]: ts=2024-03-27T10:20:57.944Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:21:12 mail2. prometheus[892]: ts=2024-03-27T10:21:12.946Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:21:27 mail2. prometheus[892]: ts=2024-03-27T10:21:27.947Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:21:42 mail2.*** prometheus[892]: ts=2024-03-27T10:21:42.949Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
I hope you can help me with these problems.
Have the same issue, and I suspect its being caused by the logrotate config, but haven't worked out how to fix yet. Restarting the rsyslog service gets logging going back to carbonio.log rather than syslog, until midnight.
Updated /etc/logrotate.d/carbonio as follows, logging continues working as expected now (do same for /var/log/carbonio-stats.log section)...
/var/log/carbonio.log {
daily
missingok
notifempty
create 0644 syslog adm
dateext
postrotate
[ -x /usr/lib/rsyslog/rsyslog-rotate ] && /usr/lib/rsyslog/rsyslog-rotate || true
su - zextras -c "/opt/zextras/bin/zmconfigdctl restart" > /dev/null 2>&1 || true
endscript
compress
rtate 30
}
Did you get help for the problem?
How did you solve it?
//Sigtrap
1. After logrotate (at 00:00), the system stops writing to the /var/log/carbonio.log file. The file occupies 0 bytes, but at the same time all entries in the journalctl go as usual. If we do not reboot the system, then at the end of the day an empty daily report comes. Also the log archive (.gz) is not created.
I solved this problem by changing the logrotate config
Good solution from @anahuac
In /etc/logrotate.d/carbonio replace the two lines (under carbonio.log and carbonio-stats.log):
kill -HUP `cat /var/run/syslog*.pid 2> /dev/null` 2> /dev/null || true
su - zextras -c "/opt/zextras/bin/zmconfigdctl restart" > /dev/null 2>&1 || true
to:
[ -x /usr/lib/rsyslog/rsyslog-rotate ] && /usr/lib/rsyslog/rsyslog-rotate || true su - zextras -c "/opt/zextras/bin/zmconfigdctl restart" > /dev/null 2>&1 || true
3 - save the file
4 - run:
/usr/lib/rsyslog/rsyslog-rotate ; su - zextras -c "/opt/zextras/bin/zmconfigdctl restart"
Also, this problem remained after updating to 24.5.0. With this service we have to do more and more manual work 🙁