carbonio-message-di...
 
Notifications
Clear all

carbonio-message-dispatcher.service timeouts way too long (upwards of 4 minutes during reboot)

6 Posts
5 Users
0 Reactions
925 Views
(@bloodyiron)
New Member
Joined: 1 year ago
Posts: 10
Topic starter  

So I have a Carbonio CE instance on an Ubuntu 24.04 VM that I installed via the installation script.

I've noticed that the Daemon located at /etc/systemd/system/multi-user.target.wants/carbonio-message-dispatcher.service takes WAYYYY to long to stop itself during rebooting (nicely) the system. As in, it ALWAYS times out and it ALWAYS takes 4 minutes to do so.

When rebooting a system that takes tens of seconds to do so, a 4 minute duration is problematic.

I've manually adjusted the service unit file to be lower, but this keeps getting undone via Carbonio updates. So honestly we need this service to be handled better.

1. The timeouts need to be much shorter.

2. The software itself needs to ACTUALLY nicely stop itself when told to do so, instead of just ignoring it and the timeout happening every time to forcefully end it.

It DOES NOT take 4 minutes to flush E-Mail messages from such a service. It's text messaging, it's not a lot of data, and this is a very quiet instance (currently).


   
Quote
 Hugo
(@hugo)
New Member
Joined: 12 months ago
Posts: 8
 

In this post: this post you can find a workaround to fix the problem, but only for manual shutdown:

systemctl stop carbonio-message-dispatcher.service ; reboot

I know it doesn't apply to all cases, but at least it's something until the definitive solution arrives.


   
ReplyQuote
 Klug
(@klug)
Trusted Member
Joined: 15 years ago
Posts: 83
 

Same issue on a "standard" installation (not using the script) on 24.04 with Carbonio CE 25.9 (that doesn't work anyway).


   
ReplyQuote
(@bloodyiron)
New Member
Joined: 1 year ago
Posts: 10
Topic starter  

Posted by: @hugo

In this post: this post you can find a workaround to fix the problem, but only for manual shutdown:

systemctl stop carbonio-message-dispatcher.service ; reboot

I know it doesn't apply to all cases, but at least it's something until the definitive solution arrives.

Well as I mentioned I've manually edited the unit file to just have shorter timeouts, indeed until the appropriate solution gets upstreamed.

 


   
ReplyQuote
(@dosnso)
New Member
Joined: 3 years ago
Posts: 12
 

Posted by: @dosnso

after upgrade from 25.3 to 25.9 , and ubuntu 22.04 to24.04 didn'twork chat.
message-dispatcher starting with fails

Spoiler
journalctl -xeu carbonio-message-dispatcher.service
окт 13 00:27:37 server.carbonio mongooseimctl[104858]: Root: /opt/zextras/common/lib/mongooseim
окт 13 00:27:37 server.carbonio mongooseim[104884]: Starting up
окт 13 00:27:37 server.carbonio mongooseimctl[104858]: Exec: /opt/zextras/common/lib/mongooseim/erts-14.2.5.2/bin/erlexec -noshell -noinput +Bd -boot /opt/zextras/common/lib/mongooseim/releases/6.2.1/start -embedded -config /etc/carbonio/message-dispatcher/app.config -args_file /etc/carbonio/message-dispatcher/vm.args -args_file /etc/carbonio/message-dispatcher/vm.dist.args -- foreground
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:39.358775+00:00 level=error what=rdbms_connection_attempt_error pid=<0.489.0> at=mongoose_rdbms:connect/4:941 sleep_for=2 backend=pgsql error={error,econnrefused}
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:39.363666+00:00 level=error reason=econnrefused pid=<0.490.0> at=gen_server:error_info/8:1391 client_info="{<0.489.0>,{'wpool_pool-mongoose_wpool$rdbms$global$default-1',[{code_server,call,1,[{file,\"code_server.erl\"},{line,150}]},{error_handler,undefined_function,3,[{file,\"error_handler.erl\"},{line,40}]},{logger_backend,do_apply_filters,4,[{file,\"logger_backend.erl\"},{line,101}]},{logger_backend,apply_filters,4,[{file,\"logger_backend.erl\"},{line,86}]},{logger_backend,log_allowed,3,[{file,\"logger_backend.erl\"},{line,32}]},{mongoose_rdbms,connect,4,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,941}]},{mongoose_rdbms,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,665}]},{wpool_process,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/_build/default/lib/worker_pool/src/wpool_process.erl\"},{line,80}]}]}}" last_message="{command,epgsql_cmd_connect,#{port => 20000,host => \"127.0.0.1\",password => #Fun<epgsql_cmd_connect.0.87005817>,username => \"carbonio-message-dispatcher-db\",codecs => [{mongoose_rdbms_pgsql_codec_boolean,[]}],database => \"carbonio-message-dispatcher-db\"}}" state={state,undefined,undefined,<<>>,undefined,on_message,undefined,{[],[]},undefined,undefined,undefined,undefined,[],information_redacted,[],undefined,undefined,undefined,undefined,undefined} log= name=<0.490.0> label={gen_server,terminate}
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:39.391541+00:00 level=error pid=<0.490.0> at=proc_lib:crash_report/4:584 report="[[{initial_call,{epgsql_sock,init,['Argument__1']}},{pid,<0.490.0>},{registered_name,[]},{error_info,{exit,econnrefused,[{gen_server,handle_common_reply,8,[{file,\"gen_server.erl\"},{line,1226}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,241}]}]}},{ancestors,['wpool_pool-mongoose_wpool$rdbms$global$default-1','wpool_pool-mongoose_wpool$rdbms$global$default-process-sup','mongoose_wpool$rdbms$global$default',mongoose_wpool_rdbms_sup,mongoose_wpool_sup,ejabberd_sup,<0.285.0>]},{message_queue_len,0},{messages,[]},{links,[<0.489.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,6772},{stack_size,28},{reductions,29818}],[]]" label={proc_lib,crash}
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: 00:27:39.383 [error] gen_server <0.490.0> terminated with reason: econnrefused
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: 00:27:39.391 [error] CRASH REPORT Process <0.490.0> with 0 neighbours exited with reason: econnrefused in gen_server:handle_common_reply/8 line 1226
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:41.391637+00:00 level=error what=rdbms_connection_attempt_error pid=<0.489.0> at=mongoose_rdbms:connect/4:941 sleep_for=4 backend=pgsql error={error,econnrefused}
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:41.391781+00:00 level=error reason=econnrefused pid=<0.494.0> at=gen_server:error_info/8:1391 client_info="{<0.489.0>,{'wpool_pool-mongoose_wpool$rdbms$global$default-1',[{gen,do_call,4,[{file,\"gen.erl\"},{line,240}]},{gen_server,call,3,[{file,\"gen_server.erl\"},{line,415}]},{epgsql,call_connect,2,[{file,\"/tmp/package/src/MongooseIM-6.2.1/_build/default/lib/epgsql/src/epgsql.erl\"},{line,207}]},{mongoose_rdbms_pgsql,connect,2,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms_pgsql.erl\"},{line,48}]},{mongoose_rdbms,connect,4,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,933}]},{mongoose_rdbms,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,665}]},{wpool_process,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/_build/default/lib/worker_pool/src/wpool_process.erl\"},{line,80}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,980}]}]}}" last_message="{command,epgsql_cmd_connect,#{port => 20000,host => \"127.0.0.1\",password => #Fun<epgsql_cmd_connect.0.87005817>,username => \"carbonio-message-dispatcher-db\",codecs => [{mongoose_rdbms_pgsql_codec_boolean,[]}],database => \"carbonio-message-dispatcher-db\"}}" state={state,undefined,undefined,<<>>,undefined,on_message,undefined,{[],[]},undefined,undefined,undefined,undefined,[],information_redacted,[],undefined,undefined,undefined,undefined,undefined} log= name=<0.494.0> label={gen_server,terminate}
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:41.392475+00:00 level=error pid=<0.494.0> at=proc_lib:crash_report/4:584 report="[[{initial_call,{epgsql_sock,init,['Argument__1']}},{pid,<0.494.0>},{registered_name,[]},{error_info,{exit,econnrefused,[{gen_server,handle_common_reply,8,[{file,\"gen_server.erl\"},{line,1226}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,241}]}]}},{ancestors,['wpool_pool-mongoose_wpool$rdbms$global$default-1','wpool_pool-mongoose_wpool$rdbms$global$default-process-sup','mongoose_wpool$rdbms$global$default',mongoose_wpool_rdbms_sup,mongoose_wpool_sup,ejabberd_sup,<0.285.0>]},{message_queue_len,0},{messages,[]},{links,[<0.489.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,4185},{stack_size,28},{reductions,30728}],[]]" label={proc_lib,crash}
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: 00:27:41.391 [error] gen_server <0.494.0> terminated with reason: econnrefused
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: 00:27:41.392 [error] CRASH REPORT Process <0.494.0> with 0 neighbours exited with reason: econnrefused in gen_server:handle_common_reply/8 line 1226
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.338158+00:00 level=critical what=app_failed_to_start reason="{timeout,{gen_server,call,[mongoose_wpool_rdbms_mgr,{start_pool,global,default,[{workers,10}],#{driver => pgsql,port => 20000,host => \"127.0.0.1\",password => \"b2b4f3143cede5ddabfb6f88d32b3d3e\",username => \"carbonio-message-dispatcher-db\",database => \"carbonio-message-dispatcher-db\",keepalive_interval => 30,max_start_interval => 30,query_timeout => 5000}}]}}" pid=<0.285.0> at=ejabberd_app:start/2:48 stacktrace="gen_server:call/2:404 mongoose_wpool:start/7:162 mongoose_wpool:'-start_configured_pools/3-lc$^1/1-1-'/1:122 mongoose_wpool:'-start_configured_pools/3-lc$^1/1-1-'/1:122 ejabberd_app:do_start/0:70 ejabberd_app:start/2:44 application_master:start_it_old/4:293"
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.340851+00:00 level=error pid=<0.425.0> at=supervisor:handle_info/2:623 unstructured_log="Supervisor received unexpected message: {'ETS-TRANSFER',mongoose_wpool,\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: <0.285.0>,undefined}\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: "
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.340990+00:00 level=error pid=<0.418.0> at=supervisor:handle_info/2:623 unstructured_log="Supervisor received unexpected message: {'ETS-TRANSFER',mongoose_wpool_http,\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: <0.285.0>,testing}\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: "
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.341331+00:00 level=error pid=<0.418.0> at=supervisor:handle_info/2:623 unstructured_log="Supervisor received unexpected message: {'ETS-TRANSFER',prepared_statements,\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: <0.285.0>,testing}\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: "
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.341065+00:00 level=error pid=<0.284.0> at=proc_lib:crash_report/4:584 report="[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.284.0>},{registered_name,[]},{error_info,{exit,{bad_return,{{ejabberd_app,start,[normal,[]]},{'EXIT',{timeout,{gen_server,call,[mongoose_wpool_rdbms_mgr,{start_pool,global,default,[{workers,10}],#{driver => pgsql,port => 20000,host => \"127.0.0.1\",password => \"b2b4f3143cede5ddabfb6f88d32b3d3e\",username => \"carbonio-message-dispatcher-db\",database => \"carbonio-message-dispatcher-db\",keepalive_interval => 30,max_start_interval => 30,query_timeout => 5000}}]}}}}},[{application_master,init,4,[{file,\"application_master.erl\"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,241}]}]}},{ancestors,[<0.283.0>]},{message_queue_len,0},{messages,[]},{links,[<0.283.0>,<0.285.0>,<0.44.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,1598},{stack_size,28},{reductions,156}],[]]" label={proc_lib,crash}
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.341792+00:00 level=notice pid=<0.44.0> at=application_controller:info_exited/3:2142 report="[{application,mongooseim},{exited,{bad_return,{{ejabberd_app,start,[normal,[]]},{'EXIT',{timeout,{gen_server,call,[mongoose_wpool_rdbms_mgr,{start_pool,global,default,[{workers,10}],#{driver => pgsql,port => 20000,host => \"127.0.0.1\",password => \"b2b4f3143cede5ddabfb6f88d32b3d3e\",username => \"carbonio-message-dispatcher-db\",database => \"carbonio-message-dispatcher-db\",keepalive_interval => 30,max_start_interval => 30,query_timeout => 5000}}]}}}}}},{type,permanent}]" label={application_controller,exit}
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: Runtime terminating during boot (terminating)
окт 13 00:27:44 server.carbonio mongooseimctl[104858]:
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: Crash dump is being written to: /var/log/carbonio/message-dispatcher/erl_crash.dump...done
окт 13 00:27:44 server.carbonio systemd[1]: carbonio-message-dispatcher.service: Main process exited, code=exited, status=1/FAILURE


   
ReplyQuote
(@jajaish)
Active Member
Joined: 2 years ago
Posts: 26
 

I think I can explain what’s happening during the shutdown — might be the same root cause as the startup crash above

 

I hit this same issue on my own Carbonio CE install (22.04, current packages as of July 2026) and dug into the logs during an actual reboot. Here’s what it looks like is happening, though I’m reasoning from the logs rather than the actual source code, so happy to be corrected if I’m misreading this:

 

When the service is told to stop, it seems to do two things at once:

 

1. The little control script (mongooseimctl) appears to just sit there counting seconds, waiting for the main program to finish on its own — 1 second, 2 seconds, 3 seconds… all the way up.

2. Meanwhile, the main program itself looks like it’s stuck trying to reconnect to its own database — over and over, 32 separate attempts in my case — because the database has already stopped (or hasn’t started yet, on the startup side). It doesn’t seem to recognize that a shutdown is happening — it just keeps retrying as if this were a normal, temporary network hiccup, until systemd eventually gives up waiting and force-kills everything.

 

If I’m reading this right, it looks like the same underlying issue as dosNSO’s startup crash further up this thread — just showing up on the other end of the service’s life. On startup, if the database isn’t ready yet, the whole thing crashes. On shutdown, if the database goes away first, it seems to just keep trying to reconnect instead of recognizing it should stop.

 

If that’s actually what’s going on, the real fix might be less about shortening the timeout (which would just make the failure happen faster, not stop it happening) and more about making sure this service properly waits for its database on startup, and stops trying to reach it as soon as a shutdown begins.

 

Could well be missing something here, so if anyone more familiar with the internals can confirm or correct this, that’d be great.


   
ReplyQuote