[Solved] Certbot runtime error

@sharif 

I downgraded according to this link https://community.zextras.com/forum/postid/12466/

As you can see below it didn't work.
I'm running version 26.6.0 on RHEL8.

Downgraded:
carbonio-certbot-2.11.1-1.el8.x86_64

Complete!
[root@smtp ~]# su - zextras
[zextras@smtp ~]$ certbot -v certificates
An unexpected error occurred:
KeyError: 'manual'
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-7z3hrcu1/log or re-run Certbot with -v for more details.
[zextras@smtp ~]$ certbot renew
An unexpected error occurred:
KeyError: 'manual'
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-s_e0anoh/log or re-run Certbot with -v for more details.
[zextras@smtp ~]$

@hwoffinden 

Thanks for testing and reporting back. The downgrade cleared the pyOpenSSL error, and on RHEL8 it has now surfaced a second, separate issue: KeyError: 'manual' means certbot started with an empty plugin registry, so its own built-in "manual" plugin wasn't found.

What happens that carbonio-certbot runs the system python3.8 against a bundled environment under /opt/zextras/common/certbot. Moving between builds (2.11.1-3 to 2.11.1-1) doesn't clean the previous build's environment and dependencies. Could you please try followings:

Note: please take backup/snapshot of server before executing/modifying any config or component.

systemctl stop carbonio-certbot.timer carbonio-certbot.service
rm -rf /opt/zextras/common/certbot/lib /opt/zextras/common/certbot/lib64
dnf reinstall -y carbonio-certbot
systemctl start carbonio-certbot.timer

The reinstall lays the 2.11.1-1 build back down cleanly, so your version lock can stay in place. If dnf reports the lock blocking the reinstall, lift it with "sudo dnf versionlock delete carbonio-certbot", run the reinstall, then re-add it.

Verify as the zextras user:

su - zextras -c 'certbot certificates'

Let me know how it goes.

Posted by: @sharif

↑

@hwoffinden 

Thanks for testing and reporting back. The downgrade cleared the pyOpenSSL error, and on RHEL8 it has now surfaced a second, separate issue: KeyError: 'manual' means certbot started with an empty plugin registry, so its own built-in "manual" plugin wasn't found.

What happens that carbonio-certbot runs the system python3.8 against a bundled environment under /opt/zextras/common/certbot. Moving between builds (2.11.1-3 to 2.11.1-1) doesn't clean the previous build's environment and dependencies. Could you please try followings:

Note: please take backup/snapshot of server before executing/modifying any config or component.

systemctl stop carbonio-certbot.timer carbonio-certbot.service
rm -rf /opt/zextras/common/certbot/lib /opt/zextras/common/certbot/lib64
dnf reinstall -y carbonio-certbot
systemctl start carbonio-certbot.timer

The reinstall lays the 2.11.1-1 build back down cleanly, so your version lock can stay in place. If dnf reports the lock blocking the reinstall, lift it with "sudo dnf versionlock delete carbonio-certbot", run the reinstall, then re-add it.

Verify as the zextras user:

su - zextras -c 'certbot certificates'

Let me know how it goes.

That worked!

Thank you very much for your help.

@sharif 
Can also confirm that downgrading, versionlock, then deleting dirs and reinstallation of carbonio-certbot-2.11.1-1.el8
worked.
I had to add "chown -R zextras:zextras /opt/zextras/common/etc/letsencrypt to allow a successful renewal

* backup of system

* $ sudo dnf downgrade -y carbonio-certbot-2.11.1-1.el8

* $ sudo dnf install python3-dnf-plugin-versionlock.noarch

*$ sudo dnf versionlock add carbonio-certbot

* $ sudo systemctl stop carbonio-certbot.timer carbonio-certbot.service

* $ sudo rm -rf /opt/zextras/common/certbot/lib /opt/zextras/common/certbot/lib64

* $ sudo dnf reinstall carbonio-certbot-2.11.1-1.el8

* $ sudo chown -R zextras:zextras /opt/zextras/common/etc/letsencrypt

* $ su - zextras

* $ certbot renew

Thanks!

Gruesse
Michael

@reset11

At this point I could say Certbot renews as the zextras user, so if /opt/zextras/common/etc/letsencrypt has ended up root-owned for any reason, the renewal can't write the new cert and fails. Your chown -R zextras:zextras gives the directory back to the account that actually runs the renewal.

When someone faces this kind of situation could try checking:

find /opt/zextras/common/etc/letsencrypt \! -user zextras \! -type l

If that lists any files,where the owner isn't the zextras uid, then should run:

chown -R zextras:zextras /opt/zextras/common/etc/letsencrypt

That's all I can say for now!

Posted by: @sharif

↑

@reset11

At this point I could say Certbot renews as the zextras user, so if /opt/zextras/common/etc/letsencrypt has ended up root-owned for any reason, the renewal can't write the new cert and fails. Your chown -R zextras:zextras gives the directory back to the account that actually runs the renewal.

When someone faces this kind of situation could try checking:

find /opt/zextras/common/etc/letsencrypt \! -user zextras \! -type l

If that lists any files,where the owner isn't the zextras uid, then should run:

chown -R zextras:zextras /opt/zextras/common/etc/letsencrypt

That's all I can say for now!

The path specified:

find /opt/zextras/common/etc/letsencrypt \! -user zextras \! -type l

doesn't exist in my installation, but the following does:

find /opt/zextras/common/certbot/etc/letsencrypt \! -user zextras \! -type l

Just wondering if my setup is different.

Regards,
Henrik

@hwoffinden 

Of course you are right - I should have copied the path - not typed it - but was on a different machine...

The correct path:
/opt/zextras/common/certbot/etc/letsencrypt

Thanks for pointing that out!

Gruesse

Michael