carbonio-auth servi...
 
Notifications
Clear all

carbonio-auth service missing entirely — no package, no registration, no systemd unit — causing 502 on webmail login

5 Posts
3 Users
0 Reactions
227 Views
(@jajaish)
Active Member
Joined: 2 years ago
Posts: 26
Topic starter  

Running Carbonio CE, carbonio-core 4.5.12-1jammy, Ubuntu 22.04.5.

Webmail/admin login fails with 502 Bad Gateway on GET /zx/login/v3/auth/config. Traced this to the carbonio-proxy upstream carbonio-auth, which routes to 127.78.0.1:20008 (an Envoy sidecar). The sidecar itself is up and listening, but nothing is registered behind it.

What I've confirmed:

  • consul catalog services shows no carbonio-auth entry at all (not unhealthy — completely absent)
  • No .hcl file for it exists in /etc/zextras/service-discover/ (only carbonio-proxy.hcl references the name, as a destination)
  • No systemd unit carbonio-auth-sidecar.service exists anywhere on disk
  • dpkg -S carbonio-auth-sidecar returns no matching package
  • apt-cache search carbonio | grep auth and apt list -a carbonio-auth* show only carbonio-auth-ui (a frontend module), no backend service
  • Not served directly by mailboxd either (curl localhost:8080/zx/login/v3/auth/config → 404)
  • No trace of any process ever attempting to register this service name in mailbox.log
  • pending-setups -a runs clean, nothing pending
  • Full zmcontrol restart and individual service restarts don't resolve it
  • Repointing the proxy upstream to carbonio-user-management (the service that appears to actually handle auth-related SOAP requests) also didn't resolve it — reverted that change

What's healthy: everything else — mail delivery, IMAP, SMTP, LDAP, MTA, proxy itself, mailbox backend. This is isolated to the webmail/admin login flow specifically.

Found one other user with the exact same symptom (community.zextras.com/forum/carbonio-general-thread/carbonio-auth-problem/, Feb 2024) with no public resolution shown.

Is carbonio-auth supposed to be provided by a specific package? Was it possibly deprecated/replaced by carbonio-user-management in a later version and this proxy template just wasn't updated to match? Any pointers appreciated.


   
Quote
(@herbert)
New Member
Joined: 1 month ago
Posts: 1
 
I had the exact same problem after upgrading to 26.6.0. Two packages were missing from the upgrade dependency chain:

  1. Install missing packages

apt install -y carbonio-catalog carbonio-user-management

Then run pending-setups -a (needs the cluster credential password) — you may need to run it twice until all scripts complete.

  1. Nginx workaround for dead upstreams

The zx (port 8742) and carbonio-auth (port 20008) upstreams are legacy — no service listens there anymore. The admin/login apps call /zx/login/v1/config and /zx/auth/supported which time out. Add these location blocks before the existing location ^~ /zx/ in your admin nginx config (in both .admin and .admin.default files under /opt/zextras/conf/nginx/includes/):

location ^~ /zx/login/v1/config
{
    return 200 '{"adminConsolePublicUrl":"https://your-server:6071","publicUrl":"https://your-server","zimbraDomainName":"your-domain","authMethods":["zimbra"]}';
}
location ^~ /zx/auth/supported
{
    return 200 '{"maxApiVersion":2,"authMethods":["zimbra"]}';
}

Then reload nginx:

su - zextras -c 'zmproxyctl reload'

  1. i18n fix (if admin login shows 404 for locale files)

ln -sf  removed link  
 removed link 
 
ln -sf /opt /zextras /admin /login-i18n removed link    /opt /zextras /admin /login-i18n removed link

Root cause: Zextras' upgrade path doesn't pull in carbonio-catalog or carbonio-user-management as dependencies. The nginx templates still reference dead upstreams from the old architecture (pre-envoy). This should be fixed by Zextras in the upgrade scripts and package dependencies. Hope this saves you the hours of digging it took me!

---

Best Regards
Herbert

This post was modified 1 month ago 2 times by herbert

   
ReplyQuote
(@jajaish)
Active Member
Joined: 2 years ago
Posts: 26
Topic starter  

Thanks for the detailed writeup, Herbert — very helpful starting point. I dug into this further and found I’m in a slightly different situation than the “missing packages” scenario:

 

carbonio-catalog and carbonio-user-management are both already installed (apt-get install -s confirms “already the newest version” for both)

Their service-discover registration files exist and are enabled (/etc/zextras/service-discover/carbonio-catalog.hcl, carbonio-user-management.hcl)

Their pending-setups scripts show as already successfully completed: /etc/zextras/pending-setups.d/done/421-carbonio-catalog.sh and done/121-carbonio-user-management.sh

But there’s no systemd unit for either service, and nothing is listening on the address/port their own .hcl file specifies (confirmed carbonio-catalog should be at 127.78.0.28:10000 — nothing there)

 

So the setup appears to have run successfully at some point, but the actual services aren’t running now and haven’t self-recovered. Running pending-setups -a again shows “no pending setups to run” — nothing left for it to do, since these are already marked complete.


   
ReplyQuote
(@jajaish)
Active Member
Joined: 2 years ago
Posts: 26
Topic starter  

Following up on my earlier post — this resolved on its own after running a routine apt-get upgrade today that included libc6 (2.35-0ubuntu3.132.35-0ubuntu3.14) and related core library packages (libc-bin, libc-dev-bin, libc-devtools, libc6-dev, locales, locales-all). During that upgrade, needrestart prompted to restart a batch of services including carbonio-catalog.service and carbonio-user-management.service — services that, before this, had no systemd unit file at all on this system (confirmed via systemctl status returning nothing, and nothing listening on their expected ports).

 

After the upgrade, both now show as genuinely active (running), with real processes, “catalog ready”/“user-management ready” status messages, and confirmed listening on their expected addresses (carbonio-catalog at 127.78.0.28:10000, matching its own .hcl service-discovery config).

 

So it looks like whatever was missing (proper systemd unit registration for these two components) got installed/corrected as a dependency of this later package update, separate from the original install path. Didn’t need the pending-setups -a re-run or any manual intervention in the end — just needed a later upgrade cycle to catch up.

 

Environment: carbonio-core and related packages current as of 2026-07-28, Ubuntu 22.04.5.


   
ReplyQuote
(@abalamut)
New Member
Joined: 3 days ago
Posts: 4
 

I can confirm that this issue is still present in Carbonio CE 26.6.2 on Ubuntu 24.04 (Noble).

Environment:

  • Carbonio Release 26.6.2

  • Ubuntu 24.04

  • carbonio-core 4.5.14-2noble

  • carbonio-proxy 4.14.4-1noble

  • carbonio-auth-ui 1.6.1-1ubuntu

  • carbonio-catalog 0.5.3-1noble

  • carbonio-user-management 1.2.1-1noble

Both carbonio-catalog and carbonio-user-management are installed and their systemd services are running.

However, web login does not work.

The login page itself loads normally, but any login attempt silently fails.

The problem can be reproduced directly:

GET /zx/login/v3/config
GET /zx/login/v3/account
GET /zx/login/v3/auth/config
GET /zx/auth/supported

All of these endpoints return HTTP 502:

HTTP ERROR 502
Connection to Upstream is Refused
Cannot connect to the upstream server

Nginx routes these requests to:

carbonio-auth -> 127.78.0.1:20008

The Carbonio Proxy Envoy sidecar is running and listens on:

127.78.0.1:20008

The sidecar configuration contains:

destination_name   = "carbonio-auth"
local_bind_port    = 20008
local_bind_address = "127.78.0.1"

But consul catalog services contains no carbonio-auth service at all.

It contains, among others:

carbonio-catalog
carbonio-user-management
carbonio-message-dispatcher-auth
carbonio-proxy

but there is no:

carbonio-auth

A direct connection to the Envoy listener succeeds, but Envoy immediately resets it:

curl  removed link 

Connected to 127.78.0.1:20008
Recv failure: Connection reset by peer

carbonio-user-management health also confirms the authentication dependency is unhealthy:

{
  "isLive": true,
  "status": "error",
  "dependencies": [
    {"name":"database","isHealthy":true},
    {"name":"authentication_service","isHealthy":false},
    ...
  ]
}

The workaround suggested in this thread fixes the problem.

I changed the following Carbonio nginx templates:

 removed link 
 removed link 

and replaced the dead carbonio-auth proxy locations with:

location ^~ /zx/login/
{
    return 404;
}

location ^~ /zx/auth/
{
    return 404;
}

Then regenerated the proxy configuration and fully restarted Carbonio nginx.

After that:

/zx/login/v3/config       -> HTTP 404
/zx/login/v3/account      -> HTTP 404
/zx/login/v3/auth/config  -> HTTP 404
/zx/auth/supported        -> HTTP 404

and web login immediately started working again.

So this appears to be a packaging/configuration bug still present in Carbonio CE 26.6.2:

  • Carbonio Proxy still generates an upstream for carbonio-auth;

  • the proxy sidecar creates the listener on 127.78.0.1:20008;

  • but no carbonio-auth service is registered in Consul;

  • consequently the new authentication endpoints return HTTP 502 and prevent users from logging in.

Could you please confirm whether carbonio-auth has been intentionally removed/replaced, and if so, remove these obsolete nginx/service-discover upstreams from the Carbonio CE packages?

The current workaround modifies package-managed nginx templates and may therefore be lost during a future Carbonio upgrade.


   
ReplyQuote