Logging and service...
 
Notifications
Clear all

Logging and service problem on new 24.3.0 (Ubuntu 22.04)

5 Posts
3 Users
0 Likes
205 Views
(@thecube)
Joined: 7 months ago
Posts: 9
Topic starter  

After update to Ubuntu 22.04 and Carbonio 24.3.0 we faced such problems:

1. After logrotate (at 00:00), the system stops writing to the /var/log/carbonio.log file. The file occupies 0 bytes, but at the same time all entries in the journalctl go as usual. If we do not reboot the system, then at the end of the day an empty daily report comes. Also the log archive (.gz) is not created.

2. We see errors in the operation of services:

carbonio-message-dispatcher.service - Carbonio Message Dispatcher daemon
     Loaded: loaded (/lib/systemd/system/carbonio-message-dispatcher.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/carbonio-message-dispatcher.service.d
             └─override.conf
     Active: active (running) since Wed 2024-03-27 11:03:24 EET; 1h 19min ago
       Docs:  https://docs.zextras.com/ 
   Main PID: 4573 (mongooseimctl)
      Tasks: 45 (limit: 38352)
     Memory: 105.5M
        CPU: 15.425s
     CGroup: /system.slice/carbonio-message-dispatcher.service
             ├─4573 /bin/bash /opt/zextras/common/bin/mongooseimctl foreground
             ├─4583 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/beam.smp -Bd -C multi_time_warp -- -root /opt/zextras/common/lib/mongooseim -bindir /opt/zextras/common/lib/mongooseim/erts-14.1/bin ->
             ├─4619 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/epmd -daemon
             ├─4625 erl_child_setup 1024
             ├─5128 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/inet_gethost 4
             └─5129 /opt/zextras/common/lib/mongooseim/erts-14.1/bin/inet_gethost 4

мар 27 11:03:24 mail2.*** systemd[1]: Started Carbonio Message Dispatcher daemon.
мар 27 11:03:24 mail2. mongooseimctl[4583]: Root: /opt/zextras/common/lib/mongooseim
мар 27 11:03:24 mail2. mongooseimctl[4583]: Exec: /opt/zextras/common/lib/mongooseim/erts-14.1/bin/erlexec -noshell -noinput +Bd -boot /opt/zextras/common/lib/mongooseim/releases/6.2.0/start >
мар 27 11:03:25 mail2. mongooseimctl[4583]: 11:03:25.049 [error] Failed to open log file /opt/zextras/common/lib/mongooseim/log/console.log with error permission denied
мар 27 11:03:25 mail2. mongooseimctl[4583]: 11:03:25.049 [error] Failed to open log file /opt/zextras/common/lib/mongooseim/log/error.log with error permission denied
мар 27 11:03:25 mail2. mongooseimctl[4583]: 11:03:25.520 [error] Failed to open crash log file /opt/zextras/common/lib/mongooseim/log/crash.log with error: permission denied
мар 27 11:03:27 mail2. mongooseimctl[4583]: when=2024-03-27T09:03:27.502767+00:00 level=notice what=mongooseim_node_started pid=<0.293.0> at=ejabberd_app:do_start/0:83 version=6.2.0 node=mong>

 

carbonio-prometheus.service - Carbonio Prometheus service
     Loaded: loaded (/lib/systemd/system/carbonio-prometheus.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-03-27 11:02:53 EET; 1h 20min ago
   Main PID: 892 (prometheus)
      Tasks: 18 (limit: 38352)
     Memory: 1020.0M
        CPU: 51.375s
     CGroup: /system.slice/carbonio-prometheus.service
             └─892 /usr/bin/prometheus --config.file=/etc/carbonio/carbonio-prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data --storage.tsdb.retention.time=180d --enable-feature=memo>

мар 27 12:20:27 mail2. prometheus[892]: ts=2024-03-27T10:20:27.941Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:20:42 mail2. prometheus[892]: ts=2024-03-27T10:20:42.943Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:20:57 mail2. prometheus[892]: ts=2024-03-27T10:20:57.944Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:21:12 mail2. prometheus[892]: ts=2024-03-27T10:21:12.946Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:21:27 mail2. prometheus[892]: ts=2024-03-27T10:21:27.947Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>
мар 27 12:21:42 mail2.*** prometheus[892]: ts=2024-03-27T10:21:42.949Z caller=consul.go:285 level=error component="discovery manager scrape" discovery=consul config=process msg="Error retrieving>

 

I hope you can help me with these problems.

 


   
Quote
(@sidkipper)
Joined: 4 months ago
Posts: 2
 

Have the same issue, and I suspect its being caused by the logrotate config, but haven't worked out how to fix yet.  Restarting the rsyslog service gets logging going back to carbonio.log rather than syslog, until midnight.


   
ReplyQuote
(@sidkipper)
Joined: 4 months ago
Posts: 2
 

Updated /etc/logrotate.d/carbonio as follows, logging continues working as expected now (do same for /var/log/carbonio-stats.log section)...

/var/log/carbonio.log {
  daily
  missingok
  notifempty
  create 0644 syslog adm
  dateext
  postrotate
    [ -x /usr/lib/rsyslog/rsyslog-rotate ] && /usr/lib/rsyslog/rsyslog-rotate || true
    su - zextras -c "/opt/zextras/bin/zmconfigdctl restart" > /dev/null 2>&1 || true
  endscript
  compress
  rtate 30
}

   
ReplyQuote
(@sigtrap)
Joined: 7 months ago
Posts: 20
 

Did you get help for the problem?

How did you solve it?

//Sigtrap


   
ReplyQuote
(@thecube)
Joined: 7 months ago
Posts: 9
Topic starter  

@sigtrap 

1. After logrotate (at 00:00), the system stops writing to the /var/log/carbonio.log file. The file occupies 0 bytes, but at the same time all entries in the journalctl go as usual. If we do not reboot the system, then at the end of the day an empty daily report comes. Also the log archive (.gz) is not created.

I solved this problem by changing the logrotate config


   
ReplyQuote