carbonio-message-di...
 
Notifications
Clear all

carbonio-message-dispatcher.service timeouts way too long (upwards of 4 minutes during reboot)

5 Posts
4 Users
0 Reactions
134 Views
(@bloodyiron)
New Member
Joined: 2 months ago
Posts: 8
Topic starter  

So I have a Carbonio CE instance on an Ubuntu 24.04 VM that I installed via the installation script.

I've noticed that the Daemon located at /etc/systemd/system/multi-user.target.wants/carbonio-message-dispatcher.service takes WAYYYY to long to stop itself during rebooting (nicely) the system. As in, it ALWAYS times out and it ALWAYS takes 4 minutes to do so.

When rebooting a system that takes tens of seconds to do so, a 4 minute duration is problematic.

I've manually adjusted the service unit file to be lower, but this keeps getting undone via Carbonio updates. So honestly we need this service to be handled better.

1. The timeouts need to be much shorter.

2. The software itself needs to ACTUALLY nicely stop itself when told to do so, instead of just ignoring it and the timeout happening every time to forcefully end it.

It DOES NOT take 4 minutes to flush E-Mail messages from such a service. It's text messaging, it's not a lot of data, and this is a very quiet instance (currently).


   
Quote
 Hugo
(@hugo)
New Member
Joined: 2 months ago
Posts: 6
 

In this post: this post you can find a workaround to fix the problem, but only for manual shutdown:

systemctl stop carbonio-message-dispatcher.service ; reboot

I know it doesn't apply to all cases, but at least it's something until the definitive solution arrives.


   
ReplyQuote
 Klug
(@klug)
Trusted Member
Joined: 14 years ago
Posts: 77
 

Same issue on a "standard" installation (not using the script) on 24.04 with Carbonio CE 25.9 (that doesn't work anyway).


   
ReplyQuote
(@bloodyiron)
New Member
Joined: 2 months ago
Posts: 8
Topic starter  

Posted by: @hugo

In this post: this post you can find a workaround to fix the problem, but only for manual shutdown:

systemctl stop carbonio-message-dispatcher.service ; reboot

I know it doesn't apply to all cases, but at least it's something until the definitive solution arrives.

Well as I mentioned I've manually edited the unit file to just have shorter timeouts, indeed until the appropriate solution gets upstreamed.

 


   
ReplyQuote
(@dosnso)
New Member
Joined: 2 years ago
Posts: 12
 

Posted by: @dosnso

after upgrade from 25.3 to 25.9 , and ubuntu 22.04 to24.04 didn'twork chat.
message-dispatcher starting with fails

Spoiler
journalctl -xeu carbonio-message-dispatcher.service
окт 13 00:27:37 server.carbonio mongooseimctl[104858]: Root: /opt/zextras/common/lib/mongooseim
окт 13 00:27:37 server.carbonio mongooseim[104884]: Starting up
окт 13 00:27:37 server.carbonio mongooseimctl[104858]: Exec: /opt/zextras/common/lib/mongooseim/erts-14.2.5.2/bin/erlexec -noshell -noinput +Bd -boot /opt/zextras/common/lib/mongooseim/releases/6.2.1/start -embedded -config /etc/carbonio/message-dispatcher/app.config -args_file /etc/carbonio/message-dispatcher/vm.args -args_file /etc/carbonio/message-dispatcher/vm.dist.args -- foreground
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:39.358775+00:00 level=error what=rdbms_connection_attempt_error pid=<0.489.0> at=mongoose_rdbms:connect/4:941 sleep_for=2 backend=pgsql error={error,econnrefused}
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:39.363666+00:00 level=error reason=econnrefused pid=<0.490.0> at=gen_server:error_info/8:1391 client_info="{<0.489.0>,{'wpool_pool-mongoose_wpool$rdbms$global$default-1',[{code_server,call,1,[{file,\"code_server.erl\"},{line,150}]},{error_handler,undefined_function,3,[{file,\"error_handler.erl\"},{line,40}]},{logger_backend,do_apply_filters,4,[{file,\"logger_backend.erl\"},{line,101}]},{logger_backend,apply_filters,4,[{file,\"logger_backend.erl\"},{line,86}]},{logger_backend,log_allowed,3,[{file,\"logger_backend.erl\"},{line,32}]},{mongoose_rdbms,connect,4,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,941}]},{mongoose_rdbms,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,665}]},{wpool_process,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/_build/default/lib/worker_pool/src/wpool_process.erl\"},{line,80}]}]}}" last_message="{command,epgsql_cmd_connect,#{port => 20000,host => \"127.0.0.1\",password => #Fun<epgsql_cmd_connect.0.87005817>,username => \"carbonio-message-dispatcher-db\",codecs => [{mongoose_rdbms_pgsql_codec_boolean,[]}],database => \"carbonio-message-dispatcher-db\"}}" state={state,undefined,undefined,<<>>,undefined,on_message,undefined,{[],[]},undefined,undefined,undefined,undefined,[],information_redacted,[],undefined,undefined,undefined,undefined,undefined} log= name=<0.490.0> label={gen_server,terminate}
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:39.391541+00:00 level=error pid=<0.490.0> at=proc_lib:crash_report/4:584 report="[[{initial_call,{epgsql_sock,init,['Argument__1']}},{pid,<0.490.0>},{registered_name,[]},{error_info,{exit,econnrefused,[{gen_server,handle_common_reply,8,[{file,\"gen_server.erl\"},{line,1226}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,241}]}]}},{ancestors,['wpool_pool-mongoose_wpool$rdbms$global$default-1','wpool_pool-mongoose_wpool$rdbms$global$default-process-sup','mongoose_wpool$rdbms$global$default',mongoose_wpool_rdbms_sup,mongoose_wpool_sup,ejabberd_sup,<0.285.0>]},{message_queue_len,0},{messages,[]},{links,[<0.489.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,6772},{stack_size,28},{reductions,29818}],[]]" label={proc_lib,crash}
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: 00:27:39.383 [error] gen_server <0.490.0> terminated with reason: econnrefused
окт 13 00:27:39 server.carbonio mongooseimctl[104858]: 00:27:39.391 [error] CRASH REPORT Process <0.490.0> with 0 neighbours exited with reason: econnrefused in gen_server:handle_common_reply/8 line 1226
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:41.391637+00:00 level=error what=rdbms_connection_attempt_error pid=<0.489.0> at=mongoose_rdbms:connect/4:941 sleep_for=4 backend=pgsql error={error,econnrefused}
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:41.391781+00:00 level=error reason=econnrefused pid=<0.494.0> at=gen_server:error_info/8:1391 client_info="{<0.489.0>,{'wpool_pool-mongoose_wpool$rdbms$global$default-1',[{gen,do_call,4,[{file,\"gen.erl\"},{line,240}]},{gen_server,call,3,[{file,\"gen_server.erl\"},{line,415}]},{epgsql,call_connect,2,[{file,\"/tmp/package/src/MongooseIM-6.2.1/_build/default/lib/epgsql/src/epgsql.erl\"},{line,207}]},{mongoose_rdbms_pgsql,connect,2,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms_pgsql.erl\"},{line,48}]},{mongoose_rdbms,connect,4,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,933}]},{mongoose_rdbms,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/src/rdbms/mongoose_rdbms.erl\"},{line,665}]},{wpool_process,init,1,[{file,\"/tmp/package/src/MongooseIM-6.2.1/_build/default/lib/worker_pool/src/wpool_process.erl\"},{line,80}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,980}]}]}}" last_message="{command,epgsql_cmd_connect,#{port => 20000,host => \"127.0.0.1\",password => #Fun<epgsql_cmd_connect.0.87005817>,username => \"carbonio-message-dispatcher-db\",codecs => [{mongoose_rdbms_pgsql_codec_boolean,[]}],database => \"carbonio-message-dispatcher-db\"}}" state={state,undefined,undefined,<<>>,undefined,on_message,undefined,{[],[]},undefined,undefined,undefined,undefined,[],information_redacted,[],undefined,undefined,undefined,undefined,undefined} log= name=<0.494.0> label={gen_server,terminate}
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:41.392475+00:00 level=error pid=<0.494.0> at=proc_lib:crash_report/4:584 report="[[{initial_call,{epgsql_sock,init,['Argument__1']}},{pid,<0.494.0>},{registered_name,[]},{error_info,{exit,econnrefused,[{gen_server,handle_common_reply,8,[{file,\"gen_server.erl\"},{line,1226}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,241}]}]}},{ancestors,['wpool_pool-mongoose_wpool$rdbms$global$default-1','wpool_pool-mongoose_wpool$rdbms$global$default-process-sup','mongoose_wpool$rdbms$global$default',mongoose_wpool_rdbms_sup,mongoose_wpool_sup,ejabberd_sup,<0.285.0>]},{message_queue_len,0},{messages,[]},{links,[<0.489.0>]},{dictionary,[]},{trap_exit,false},{status,running},{heap_size,4185},{stack_size,28},{reductions,30728}],[]]" label={proc_lib,crash}
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: 00:27:41.391 [error] gen_server <0.494.0> terminated with reason: econnrefused
окт 13 00:27:41 server.carbonio mongooseimctl[104858]: 00:27:41.392 [error] CRASH REPORT Process <0.494.0> with 0 neighbours exited with reason: econnrefused in gen_server:handle_common_reply/8 line 1226
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.338158+00:00 level=critical what=app_failed_to_start reason="{timeout,{gen_server,call,[mongoose_wpool_rdbms_mgr,{start_pool,global,default,[{workers,10}],#{driver => pgsql,port => 20000,host => \"127.0.0.1\",password => \"b2b4f3143cede5ddabfb6f88d32b3d3e\",username => \"carbonio-message-dispatcher-db\",database => \"carbonio-message-dispatcher-db\",keepalive_interval => 30,max_start_interval => 30,query_timeout => 5000}}]}}" pid=<0.285.0> at=ejabberd_app:start/2:48 stacktrace="gen_server:call/2:404 mongoose_wpool:start/7:162 mongoose_wpool:'-start_configured_pools/3-lc$^1/1-1-'/1:122 mongoose_wpool:'-start_configured_pools/3-lc$^1/1-1-'/1:122 ejabberd_app:do_start/0:70 ejabberd_app:start/2:44 application_master:start_it_old/4:293"
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.340851+00:00 level=error pid=<0.425.0> at=supervisor:handle_info/2:623 unstructured_log="Supervisor received unexpected message: {'ETS-TRANSFER',mongoose_wpool,\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: <0.285.0>,undefined}\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: "
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.340990+00:00 level=error pid=<0.418.0> at=supervisor:handle_info/2:623 unstructured_log="Supervisor received unexpected message: {'ETS-TRANSFER',mongoose_wpool_http,\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: <0.285.0>,testing}\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: "
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.341331+00:00 level=error pid=<0.418.0> at=supervisor:handle_info/2:623 unstructured_log="Supervisor received unexpected message: {'ETS-TRANSFER',prepared_statements,\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: <0.285.0>,testing}\
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: "
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.341065+00:00 level=error pid=<0.284.0> at=proc_lib:crash_report/4:584 report="[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.284.0>},{registered_name,[]},{error_info,{exit,{bad_return,{{ejabberd_app,start,[normal,[]]},{'EXIT',{timeout,{gen_server,call,[mongoose_wpool_rdbms_mgr,{start_pool,global,default,[{workers,10}],#{driver => pgsql,port => 20000,host => \"127.0.0.1\",password => \"b2b4f3143cede5ddabfb6f88d32b3d3e\",username => \"carbonio-message-dispatcher-db\",database => \"carbonio-message-dispatcher-db\",keepalive_interval => 30,max_start_interval => 30,query_timeout => 5000}}]}}}}},[{application_master,init,4,[{file,\"application_master.erl\"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,241}]}]}},{ancestors,[<0.283.0>]},{message_queue_len,0},{messages,[]},{links,[<0.283.0>,<0.285.0>,<0.44.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,1598},{stack_size,28},{reductions,156}],[]]" label={proc_lib,crash}
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: when=2025-10-12T17:27:44.341792+00:00 level=notice pid=<0.44.0> at=application_controller:info_exited/3:2142 report="[{application,mongooseim},{exited,{bad_return,{{ejabberd_app,start,[normal,[]]},{'EXIT',{timeout,{gen_server,call,[mongoose_wpool_rdbms_mgr,{start_pool,global,default,[{workers,10}],#{driver => pgsql,port => 20000,host => \"127.0.0.1\",password => \"b2b4f3143cede5ddabfb6f88d32b3d3e\",username => \"carbonio-message-dispatcher-db\",database => \"carbonio-message-dispatcher-db\",keepalive_interval => 30,max_start_interval => 30,query_timeout => 5000}}]}}}}}},{type,permanent}]" label={application_controller,exit}
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: Runtime terminating during boot (terminating)
окт 13 00:27:44 server.carbonio mongooseimctl[104858]:
окт 13 00:27:44 server.carbonio mongooseimctl[104858]: Crash dump is being written to: /var/log/carbonio/message-dispatcher/erl_crash.dump...done
окт 13 00:27:44 server.carbonio systemd[1]: carbonio-message-dispatcher.service: Main process exited, code=exited, status=1/FAILURE


   
ReplyQuote