These are direct copies from the wiki and references to other sources, they haven't been tested yet but given then we might need to do this in 10 or 20 years I figure its best to create our own copy here. ====General Certificate Renewal==== https://www.dogtagpki.org/wiki/Certificate_Renewal Overview This document describes general certificate renewal process. To renew user certificates see User Certificate Renewal. To renew system certificates used by PKI server see System Certificate Renewal. Submitting Renewal Request PKI 10.4 or later To renew CA certificate: $ pki ca-cert-request-submit --profile caCACert --csr-file ca_signing.csr --renewal --serial 0x1 ----------------------------- Submitted certificate request ----------------------------- Request ID: 7 Type: enrollment Request Status: pending Operation Result: success To renew other certificates: $ pki ca-cert-request-submit --profile caManualRenewal --serial 0x6 --renewal ----------------------------- Submitted certificate request ----------------------------- Request ID: 28 Type: renewal Request Status: pending Operation Result: success Approving Renewal Request If the certificate renewal requires agent approval, it can be done with the following command: $ pki -d ~/.dogtag/pki-tomcat/ca/alias -c Secret.123 -n caadmin \ ca-cert-request-review 28 --action approve ------------------------------- Approved certificate request 28 ------------------------------- Request ID: 28 Type: enrollment Request Status: complete Operation Result: success Certificate ID: 0x1c See Handling Certificate Request. Retrieving Renewed Certificate To download the new certificate into a file: $ pki ca-cert-show 0x1c --output testuser.crt Changing System Date Sometimes it is necessary to change the system date in order to renew a certificate, for example to renew an already expired certificate or for testing. To change the system date, first disable NTP: $ timedatectl set-ntp false Then change the date: $ timedatectl set-time To reenable NTP: $ timedatectl set-ntp true --adjust-system-clock To check NTP status: $ ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== +chl.la 216.218.254.202 2 u 8 64 3 174.046 2.242 2.282 *utcnist2.colora .NIST. 1 u 3 64 3 164.484 10.623 3.013 ntp.newfxlabs.c .STEP. 16 u - 64 0 0.000 0.000 0.000 +103.245.79.2 118.189.138.5 2 u 9 64 3 275.932 -4.128 1.948 If the output looks like this, check the NTP configuration is located at /etc/ntp.conf. remote refid st t when poll reach delay offset jitter ============================================================================== *LOCAL(0) .LOCL. 10 l 5 64 377 0.000 0.000 0.000 To force time sync: $ ntpdate -s time.nist.gov ====System Certificate Renewal==== https://www.dogtagpki.org/wiki/System_Certificate_Renewal Overview This page describes the procedure to renew system certificates used by PKI server. Depending on the subsystems installed, PKI server uses several system certificates, for example: CA signing certificate OCSP signing certificate SSL server certificate Subsystem certificate Audit signing certificate In order for the server to operate properly all of these certificates must be valid. The CA signing certificate usually has the longest validity range (default is 20 years). The other certificates usually have shorter validity range (e.g. 2 years). To avoid service interruption any expiring certificate should be renewed while it's still valid. Note that the default PKI user certificate also has a shorter validity range. See User Certificate Renewal. Also note that the renewal procedure requires a running CA. If the expired system certificate was issued by a CA running on the same server, the server might fail to start and the certificate cannot be renewed since the CA is not available. If that happens, the system date needs to be moved back to a time where all system certificates were still valid before starting the renewal process. This document describes the renewal procedure using CLI in PKI 10.x. In PKI 9.x the renewal procedure needs to be done using legacy servlet interface, but it still follows the same general idea. Identifying Certificates to Renew First, identify the serial numbers of the certificates that are about to expire. In PKI 10.4 or later the serial numbers and the validity dates of the system certificates can be determined with the following command: $ pki-server subsystem-cert-find ca ----------------- 5 entries matched ----------------- Cert ID: signing Nickname: ca_signing Token: Internal Key Storage Token Serial number: 0x1 Issuer: CN=CA Signing Certificate, OU=pki-tomcat, O=EXAMPLE Subject: CN=CA Signing Certificate, OU=pki-tomcat, O=EXAMPLE Not valid before: 2016-12-06 12:49:43 Not valid after: 2036-12-06 12:49:43 Cert ID: ocsp_signing Nickname: ca_ocsp_signing Token: Internal Key Storage Token Serial number: 0x2 Issuer: CN=CA Signing Certificate, OU=pki-tomcat, O=EXAMPLE Subject: CN=CA OCSP Signing Certificate, OU=pki-tomcat, O=EXAMPLE Not valid before: 2016-12-06 12:49:43 Not valid after: 2018-11-26 12:49:43 Cert ID: sslserver Nickname: sslserver Token: Internal Key Storage Token Serial number: 0x3 Issuer: CN=CA Signing Certificate, OU=pki-tomcat, O=EXAMPLE Subject: CN=vm-058-091.abc.idm.lab.eng.brq.redhat.com, OU=pki-tomcat, O=EXAMPLE Not valid before: 2016-12-06 12:49:44 Not valid after: 2018-11-26 12:49:44 Cert ID: subsystem Nickname: subsystem Token: Internal Key Storage Token Serial number: 0x4 Issuer: CN=CA Signing Certificate, OU=pki-tomcat, O=EXAMPLE Subject: CN=Subsystem Certificate, OU=pki-tomcat, O=EXAMPLE Not valid before: 2016-12-06 12:49:44 Not valid after: 2018-11-26 12:49:44 Cert ID: audit_signing Nickname: ca_audit_signing Token: Internal Key Storage Token Serial number: 0x5 Issuer: CN=CA Signing Certificate, OU=pki-tomcat, O=EXAMPLE Subject: CN=CA Audit Signing Certificate, OU=pki-tomcat, O=EXAMPLE Not valid before: 2016-12-06 12:49:44 Not valid after: 2018-11-26 12:49:44 PKI 10.4 or later In PKI 10.4 it can be done with the following commands: $ pki ca-cert-request-submit --profile caManualRenewal --serial 0x2 --renewal # OCSP signing $ pki ca-cert-request-submit --profile caManualRenewal --serial 0x3 --renewal # SSL server $ pki ca-cert-request-submit --profile caManualRenewal --serial 0x4 --renewal # subsystem $ pki ca-cert-request-submit --profile caManualRenewal --serial 0x5 --renewal # audit signing Approving Renewal Requests As the CA agent, approve each renewal request using the request ID: $ pki -d ~/.dogtag/pki-tomcat/ca/alias -n caadmin -c Secret.123 \ ca-cert-request-review 7 --action approve $ pki -d ~/.dogtag/pki-tomcat/ca/alias -n caadmin -c Secret.123 \ ca-cert-request-review 8 --action approve $ pki -d ~/.dogtag/pki-tomcat/ca/alias -n caadmin -c Secret.123 \ ca-cert-request-review 9 --action approve $ pki -d ~/.dogtag/pki-tomcat/ca/alias -n caadmin -c Secret.123 \ ca-cert-request-review 10 --action approve Each approval will generate a new certificate with a new serial number. See also Approving Renewal Request. Retrieving Renewed Certificates Download the renewed certificates into files using the new serial numbers: $ pki ca-cert-show 0x7 --output ca_ocsp_signing.crt $ pki ca-cert-show 0x8 --output sslserver.crt $ pki ca-cert-show 0x9 --output subsystem.crt $ pki ca-cert-show 0xa --output ca_audit_signing.crt See also Retrieving Renewed Certificate. Updating System Certificates The new certificates need to be imported into the NSS database and the copy of the certificates in the CS.cfg need to be updated as well. While NSS should be able to handle multiple certificates (both old and new) in the NSS database it is recommended that the old one is removed. To prevent NSS database or CS.cfg corruption, shutdown the server before making the changes. PKI 10.4 or later First, shutdown the server: $ systemctl stop pki-tomcatd@pki-tomcat.service Update the NSS database and CS.cfg with the following commands: $ pki-server subsystem-cert-update ca ocsp_signing --cert ca_ocsp_signing.crt $ pki-server subsystem-cert-update ca sslserver --cert sslserver.crt $ pki-server subsystem-cert-update ca subsystem --cert subsystem.crt $ pki-server subsystem-cert-update ca audit_signing --cert ca_audit_signing.crt Finally, restart the server: $ systemctl start pki-tomcatd@pki-tomcat.service ====User Certificate Renewal==== https://www.dogtagpki.org/wiki/User_Certificate_Renewal Overview This document describes the process to renew a user certificate that can be used to authenticate against PKI server. Submitting Renewal Request Renewing valid user certificate To renew a user certificate using the certificate itself for authentication, the certificate must still be valid and stored in the client NSS database (see Retrieving Certificate): $ pki -c Secret.123 client-cert-find ---------------------- 1 certificate(s) found ---------------------- Serial Number: 0x7 Nickname: testuser Subject DN: UID=testuser Issuer DN: CN=CA Signing Certificate,O=EXAMPLE ---------------------------- Number of entries returned 1 ---------------------------- Submit the renewal request over SSL connection using the client certificate for authentication: $ pki -U https://: -c -n \ client-cert-request --profile For example: $ pki -U https://server.example.com:8443 -c Secret.123 -n testuser client-cert-request uid=testuser \ --profile caManualRenewal See Certificate Profiles. Renewing expired user certificate If the user certificate has already expired, use the general Certificate Renewal procedure. Approving Renewal Request If the certificate renewal requires agent approval, it can be done with the following command: $ pki -d ~/.dogtag/pki-tomcat/ca/alias -c Secret.123 -n caadmin \ ca-cert-request-review 28 --action approve ------------------------------- Approved certificate request 28 ------------------------------- Request ID: 28 Type: enrollment Request Status: complete Operation Result: success Certificate ID: 0x1c See Handling Certificate Request. Assigning Renewed Certificate to User As the PKI admin assign the new certificate to the user. PKI 10.4 or later Assign the new certificate to the user with this command: $ pki -d ~/.dogtag/pki-tomcat/ca/alias -c Secret.123 -n caadmin ca-user-cert-add testuser --serial 0x1c Then delete the old certificate from the user. In PKI 10.4 it can be done with the following command: $ pki -d ~/.dogtag/pki-tomcat/ca/alias -c Secret.123 -n caadmin ca-user-cert-del testuser --serial 0x7 Retrieving Renewed Certificate As the user, replace the certificate in the client NSS database. PKI 10.4 or later In PKI 10.4 it can be done with the following command: $ pki -c Secret.123 client-cert-import testuser --serial 0x1c --overwrite ====Offline System Certificate Renewal==== https://raw.githubusercontent.com/dogtagpki/pki/master/docs/admin/Offline_System_Certificate_Renewal.md ## Overview PKI server provides a mechanism to recover from expired system certificates. This mechanism can also be used to renew the certificates before they expire. There are 2 ways to renew the certs 1. [Automated Renewal Process](#Automated-Renewal-Process) 2. [Manual Renewal Process](#Manual-Renewal-Process) It is assumed that you have the following: * Valid CA signing cert * Valid admin cert To verify these assumptions are valid: 1. List details of all system certificates. (Note down the `` of the certificates that need to be renewed) ```` # pki-server cert-find Cert ID: ca_signing Nickname: ca_signing Serial Number: 0x1 Subject DN: CN=CA Signing Certificate,OU=pki-tomcat,O=EXAMPLE Issuer DN: CN=CA Signing Certificate,OU=pki-tomcat,O=EXAMPLE Not Valid Before: Wed Dec 19 17:33:21 2018 Not Valid After: Sun Dec 19 17:33:21 2038 ```` 2. Check details of the admin certificate ```` # certutil -L \ -d \ -n Certificate: Data: Version: 3 (0x2) Serial Number: 6 (0x6) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=CA Signing Certificate,OU=pki-tomcat,O=EXAMPLE" Validity: Not Before: Sat Dec 15 02:16:26 2018 Not After : Fri Dec 04 02:16:26 2020 Subject: "CN=PKI Administrator,E=caadmin@example.com,OU=pki-tomcat,O= EXAMPLE" ```` ## Automated Renewal Process One line tool that fixes all certificates: # pki-server cert-fix \ -n \ -d \ -c One line tool to fix one particular certificate: # pki-server cert-fix --cert \ -n \ -d \ -c For all available options, you can type: $ pki-server cert-fix --help ## Manual Renewal Process ### Initialization It is recommended to run the following steps to ensure that `CS.cfg` and NSS database are synchronized and that the server can operate without any issues. 1. Disable self tests using the following command: ```` # pki-server selftest-disable ```` 2. Synchronize NSS database and CS.cfg for all system certificates that are to be renewed ```` # pki-server cert-update ```` ### Bringing up the PKI server 1. Create temp SSL certificate. The temp cert will be created in `/etc/pki//certs/sslserver.crt` ```` # pki-server cert-create sslserver --temp ```` 2. Delete the existing System cert if exist ```` # pki-server cert-del sslserver ```` 3. Import temp SSL certificate created in previous step ```` # pki-server cert-import sslserver ```` 4. Start server ```` # systemctl start pki-tomcatd@pki-tomcat ```` ### System Certificate Renewal 1. Renew required system certs using PKI tool. For **`sslserver`** cert provide the `serial number` from the **original SSL server cert** to avoid placing request for unintended cert. ```` # pki-server cert-create --renew \ -n \ -d \ -c \ \ --serial ```` **OR** using 3rd party tool, like certmonger. Please refer [certmonger manual](https://www.freeipa.org/page/Certmonger) to renew the certs. 2. Stop server to update PKI server instance to use latest renewed certs ```` # systemctl stop pki-tomcatd@pki-tomcat ```` 3. Delete the existing NSS database certs ```` # pki-server cert-del ```` 4. Import the renewed permanent certs into NSS db and update corresponding CS.cfg files ```` # pki-server cert-import ```` *Note:* Make sure the **Audit Log** has the trust flags: ***"u,u,Pu"*** by running the following command: ```` # certutil -L -d /var/lib/pki/pki-tomcat/alias/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ca_signing CTu,Cu,Cu transportCert cert-pki-tomcat KRA u,u,u storageCert cert-pki-tomcat KRA u,u,u auditSigningCert cert-pki-tomcat KRA u,u,Pu subsystem u,u,u sslserver u,u,u ca_ocsp_signing u,u,u ca_audit_signing u,u,Pu ```` 5. Enable the self test using the following command: ```` # pki-server selftest-enable ```` 6. Start server with new renewed system certificates. ```` # systemctl start pki-tomcatd@pki-tomcat ```` ====Offline expired certificate renewal for Dogtag==== https://frasertweedale.github.io/blog-redhat/posts/2019-02-28-dogtag-cert-fix.html The worst has happened. Somehow, certificate renewal didn’t happen when it should have, and now you have expired certificates. Worst, these are Dogtag system certificates; you can’t even start Dogtag to issue new ones! Unfortunately, this situation arises fairly often. Sometimes due to administrator error or extended downtime; sometimes due to bugs. These cases are notoriously difficult (and expensive) to analyse and resolve. It often involves time travel: Set the system clock to a time setting just before certificates started expiring. Fix whatever caused renewal not to work in the first place. Renew expiring certificates. Reset system clock. That is the simple case! I have seen much gnarlier scenarios. Ones where multiple times must be visited. Ones where there is no time at which all relevant certs are valid. It would be nice to avoid these scenarios, and the FreeIPA team continues to work to improve the robustness of certificate renewal. We also have a monitoring / health check solution on the roadmap, so that failure of automated renewal sets off alarms before everything else falls over. But in the meantime, customers and support are still dealing with scenarios like this. Better recovery tools are needed. And better tools are on the way! Dinesh, one of the Dogtag developers, has built a tool to simplify renewal when your Dogtag CA is offline due to expired system certificates. This post outlines what the tool is, what it does, and my first experiences using it in a FreeIPA deployment. Along the way and especially toward the end of the post, I will discuss the caveats and potential areas for improvement, and FreeIPA-specific considerations. pki-server cert-fix The tool is implemented as a subcommand of the pki-server utility–namely cert-fix (and I will use this short name throughout the post). So it is implemented in Python, but in some places it calls out to certutil or the Java parts of Dogtag via the HTTP API. The user documentation is maintained the source repository. The insight at the core of cert-fix is that even if Dogtag is not running or cannot run, we still have access to the keys needed to issue certificates. We do need to use Dogtag to properly store issued certificates (for revocation purposes) and produce an audit trail. But if needed, we can use the CA signing key to temporarily fudge the important certificates to get Dogtag running again, then re-issue expired system certificates properly. Assumptions cert-fix makes the following assumptions about your environment. If these do not hold, then cert-fix, as currently implemented, cannot do its thing. The CA signing certificate is valid. You have a valid admin or agent certificate. In a FreeIPA environment the IPA RA certificaite fulfils this role. (indirect) The LDAP server (389 DS) is operational, its certificate is valid, and Dogtag can authenticate to it. These assumptions have been made for good reasons, but there are several certificate expiry scenarios that breach them. I will discuss in detail later in the post. For now, we must accept them. What cert-fix does The cert-fix performs the following actions to renew an expired system certificate: Inspect the system and identify which system certificates need renewing. Or the certificates can be specified on the command line. If Dogtag’s HTTPS certificate is expired, use certutil commands to issue a new “temporary” certificate. The validity period is three months (from the current time). The serial number of the current (expired) HTTPS is reused (a big X.509 no-no, but operationally no big deal in this scenario). There is no audit trail and the certificate will not appear in the LDAP database. Disable the startup self-test for affected subsystems, then start Dogtag. For each target certificate, renew the certificate via API, using given credential. Validity periods and other characteristics are determined by relevant profiles. Serial numbers are chosen in the usual manner, the certificates appear in LDAP and there is an audit trail. Stop Dogtag. For each target certificate, import the new certificate into Dogtag’s NSSDB. Re-enable self-test for affected subsystems and start Dogtag. Using cert-fix There are a couple of ways to try out the tool—without waiting for certificates to expire, that is. One way is to roll your system clock forward, beyond the expiry date of one or more certificates. Another possibility is to modify a certificate profile used for a system certificate so that it will be issued with a very short validity period. I opted for the latter option. I manually edited the default profile configuration, so that Dogtag’s OCSP and HTTPS certificates would be issued with a validity period of 15 minutes. By the time I installed FreeIPA, grabbed a coffee and read a few emails, the certificates had expired. Certmonger didn’t even attempt to renew them. Dogtag was still running and working properly, but ipactl restart put Dogtag, and the whole FreeIPA deployment, out of action. I used pki-server cert-find to have a peek at Dogtag’s system certificates: [root@f29-0 ca]# pki-server cert-find Cert ID: ca_signing Nickname: caSigningCert cert-pki-ca Serial Number: 0x1 Subject DN: CN=Certificate Authority,O=IPA.LOCAL 201902271325 Issuer DN: CN=Certificate Authority,O=IPA.LOCAL 201902271325 Not Valid Before: Wed Feb 27 14:30:22 2019 Not Valid After: Mon Feb 27 14:30:22 2034 Cert ID: ca_ocsp_signing Nickname: ocspSigningCert cert-pki-ca Serial Number: 0x2 Subject DN: CN=OCSP Subsystem,O=IPA.LOCAL 201902271325 Issuer DN: CN=Certificate Authority,O=IPA.LOCAL 201902271325 Not Valid Before: Wed Feb 27 14:30:24 2019 Not Valid After: Wed Feb 27 14:45:24 2019 Cert ID: sslserver Nickname: Server-Cert cert-pki-ca Serial Number: 0x3 Subject DN: CN=f29-0.ipa.local,O=IPA.LOCAL 201902271325 Issuer DN: CN=Certificate Authority,O=IPA.LOCAL 201902271325 Not Valid Before: Wed Feb 27 14:30:24 2019 Not Valid After: Wed Feb 27 14:45:24 2019 Cert ID: subsystem Nickname: subsystemCert cert-pki-ca Serial Number: 0x4 Subject DN: CN=CA Subsystem,O=IPA.LOCAL 201902271325 Issuer DN: CN=Certificate Authority,O=IPA.LOCAL 201902271325 Not Valid Before: Wed Feb 27 14:30:24 2019 Not Valid After: Tue Feb 16 14:30:24 2021 Cert ID: ca_audit_signing Nickname: auditSigningCert cert-pki-ca Serial Number: 0x5 Subject DN: CN=CA Audit,O=IPA.LOCAL 201902271325 Issuer DN: CN=Certificate Authority,O=IPA.LOCAL 201902271325 Not Valid Before: Wed Feb 27 14:30:24 2019 Not Valid After: Tue Feb 16 14:30:24 2021 Note the Not Valid After times for the ca_ocsp_signing and sslserver certificates. These are certificates we must renew. Preparing the agent certificate The cert-fix command requires an agent certificate. We will use the IPA RA certificate. The pki-server CLI tool needs an NSSDB with the agent key and certificate. So we have to set that up. First initialise the NSSDB: [root@f29-0 ~]# mkdir ra-nssdb [root@f29-0 ~]# cd ra-nssdb [root@f29-0 ra-nssdb]# certutil -d . -N Enter a password which will be used to encrypt your keys. The password should be at least 8 characters long, and should contain at least one non-alphabetic character. Enter new password: XXXXXXXX Re-enter password: XXXXXXXX Then create a PKCS #12 file containing the required key and certificates: [root@f29-0 ra-nssdb]# openssl pkcs12 -export \ -inkey /var/lib/ipa/ra-agent.key \ -in /var/lib/ipa/ra-agent.pem \ -name "ra-agent" \ -certfile /etc/ipa/ca.crt > ra-agent.p12 Enter Export Password: Verifying - Enter Export Password: Import it into the NSSDB, and fix up trust flags on the IPA CA certificate: [root@f29-0 ra-nssdb]# pk12util -d . -i ra-agent.p12 Enter Password or Pin for "NSS Certificate DB": Enter password for PKCS12 file: pk12util: PKCS12 IMPORT SUCCESSFUL [root@f29-0 ra-nssdb]# certutil -d . -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ra-agent u,u,u Certificate Authority - IPA.LOCAL 201902271325 ,, [root@f29-0 ra-nssdb]# certutil -d . -M \ -n 'Certificate Authority - IPA.LOCAL 201902271325' \ -t CT,C,C Enter Password or Pin for "NSS Certificate DB": [root@f29-0 ra-nssdb]# certutil -d . -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ra-agent u,u,u Certificate Authority - IPA.LOCAL 201902271325 CT,C,C Running cert-fix Let’s look at the cert-fix command options: [root@f29-0 ra-nssdb]# pki-server cert-fix --help Usage: pki-server cert-fix [OPTIONS] --cert Fix specified system cert (default: all certs). -i, --instance Instance ID (default: pki-tomcat). -d NSS database location (default: ~/.dogtag/nssdb) -c NSS database password -C Input file containing the password for the NSS database. -n Client certificate nickname -v, --verbose Run in verbose mode. --debug Run in debug mode. --help Show help message. It’s not a good idea to put passphrases on the command line in the clear, so let’s write the NSSDB passphrase to a file: [root@f29-0 ra-nssdb]# cat > pwdfile.txt XXXXXXXX ^D Finally, I was ready to execute cert-fix: [root@f29-0 ra-nssdb]# pki-server cert-fix \ -d . -C pwdfile.txt -n ra-agent \ --cert sslserver --cert ca_ocsp_signing \ --verbose Running with --verbose causes INFO and higher-level log messages to be printed to the terminal. Running with --debug includes DEBUG messages. If neither of these is used, nothing is output (unless there’s an error). So I recommend running with --verbose. So, what happened? Unfortunately I ran into several issues. 389 DS not running The first issue was trivial, but likely to occur if you have to cert-fix a FreeIPA deployment. The ipactl [re]start command will shut down every component if any component failed to start. Dogtag didn’t start, therefore ipactl shut down 389 DS too. As a consequence, Dogtag failed to initialise after cert-fix started it, and the command failed. So, before running cert-fix, make sure LDAP is working properly. To start it, use systemctl instead of ipactl: # systemctl start dirsrv@YOUR-REALM Connection refused One issue I encountered was that a slow startup of Dogtag caused failure of the tool. cert-fix does not wait for Dogtag to start up properly. It just ploughs ahead—only to encounter ConnectionRefusedError. I worked around this—temporarily—by adding a sleep after cert-fix starts Dogtag. A proper fix will require a change to the code. cert-fix should perform a server status check, retrying until it succeeds or times out. TLS handshake failure The next error I encountered was a TLS handshake failure: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='f29-0.ipa.local', port=8443): Max retries exceeded with url: /ca/rest/certrequests/profiles/caManualRenewal (Caused by SSLError(SSLError(185073780, '[X 509: KEY_VALUES_MISMATCH] key values mismatch (_ssl.c:3841)'))) I haven’t worked out yet what is causing this surprising error. But I wasn’t the first to encounter it. A comment in the Bugzilla ticket indicated that the workaround was to remove the IPA CA certificate from the client NSSDB. This I did: [root@f29-0 ra-nssdb]# certutil -d . -D \ -n "Certificate Authority - IPA.LOCAL 201902271325" After this, my next attempt at running cert-fix succeeded. Results Looking at the previously expired target certificates, observe that the certificates have been updated. They have new serial numbers, and expire in 15 months: [root@f29-0 ra-nssdb]# certutil -d /etc/pki/pki-tomcat/alias \ -L -n 'Server-Cert cert-pki-ca' | egrep "Serial|Not After" Serial Number: 12 (0xc) Not After : Wed May 27 12:45:25 2020 [root@f29-0 ra-nssdb]# certutil -d /etc/pki/pki-tomcat/alias \ -L -n 'ocspSigningCert cert-pki-ca' | egrep "Serial|Not After" Serial Number: 13 (0xd) Not After : Wed May 27 12:45:28 2020 Looking at the output of getcert list for the target certificates, we see that Certmonger has not picked these up (some lines removed): [root@f29-0 ra-nssdb]# getcert list -i 20190227033149 Number of certificates and requests being tracked: 9. Request ID '20190227033149': status: CA_UNREACHABLE ca-error: Internal error stuck: no CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.LOCAL 201902271325 subject: CN=OCSP Subsystem,O=IPA.LOCAL 201902271325 expires: 2019-02-27 14:45:24 AEDT eku: id-kp-OCSPSigning [root@f29-0 ra-nssdb]# getcert list -i 20190227033152 Number of certificates and requests being tracked: 9. Request ID '20190227033152': status: CA_UNREACHABLE ca-error: Internal error stuck: no CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.LOCAL 201902271325 subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201902271325 expires: 2019-02-27 14:45:24 AEDT dns: f29-0.ipa.local key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-serverAuth Restarting Certmonger (systemctl restart certmonger) resolved the discrepancy. Finally, ipactl restart puts everything back online. cert-fix has saved the day! [root@f29-0 ra-nssdb]# ipactl restart Restarting Directory Service Starting krb5kdc Service Starting kadmin Service Starting httpd Service Starting ipa-custodia Service Starting pki-tomcatd Service Starting ipa-otpd Service ipa: INFO: The ipactl command was successful Issues and caveats Besides the issues already covered, there are several scenarios that cert-fix cannot handle. Expired CA certificate Due to the the long validity period of a typical CA certificate, the assumption that the CA certificate is valid is the safest assumption made by cert-fix. But it is not a safe assumption. The most common way this assumption is violated is with externally-signed CA certificates. For example, the FreeIPA CA in your organisation is signed by Active Directory CA, with a validity period of two years. Things get overlooked and suddenly, your FreeIPA CA is expired. It may take some time for the upstream CA administrators to issue a new certificate. In the meantime, you want to get your FreeIPA/Dogtag CA back up. Right now cert-fix doesn’t handle this scenario. I think it should. As far as I can tell, this should be straightforward to support. Unlike the next few issues… Agent certificate expiry This concerns the assumption that you have a valid agent certificate. Dogtag requires authentication to perform privilieged operations like certificate issuance. Also, the authenticated user must be included in audit events. cert-fix must issue certificates properly (with limiited temporary fudging tolerated for operational efficacy), therefore there must be an agent credential. And if your agent credential is a certificate, it must be valid. So if your agent certificate is expired, it’s Catch-22. That is why the tool, as currently implemented, must assume you have a valid, non-expired agent certificate. In some deployments the agent certificate is renewed on a different cadence from subsystem certificates. In that case, this scenario is less like to occur—but still entirely possible! The assumption is bad. In my judgement it is fairly important to find a workaround for this. One idea could be to talk directly to LDAP and set a randomly-generated password on an agent account, and use that to authenticate. After the tool exits, the passphrase is forgotten. This approach means cert-fix needs a credential and privileges to perform those operations in LDAP. Speaking of LDAP… 389 DS certificate authentication In FreeIPA deployments, Dogtag is configured to use the subsystem certificate to bind (authenticate) to the LDAP server. If the subsystem certificate is expired, 389 DS will reject the certificate; the connection fails and and Dogtag cannot start. A workaround for this may be to temporarily reconfigure Dogtag to use a password to authenticate to LDAP. Then after the new subsystem certificate was issued, it must be added to the pkidbuser entry in LDAP, and certificate authentication reinstated. This is not a FreeIPA-specific consideration. Using TLS client authentication to bind to LDAP is a supported configuration in Dogtag / RHCS. So we should probably support it in cert-fix too, somehow, since the point of the tool is to avoid complex manual procedures in recovering from expired system certificates. 389 DS service certificate expiry You know the tune by now… if this certificate is expired, Dogtag can’t talk to LDAP and can’t start, therefore a new LDAP certificate can’t be issued. Issuing a temporary certificate with the same serial number may be the best way forward here, like what we do for the Dogtag HTTPS certificate. Re-keying …is not supported. But it is a possible future enhancement Serial number reuse Re-using a serial number is prohibited by the X.509 standard. Although the temporary re-issued HTTPS certificate is supposed to be temporary, what if it did leak out? For example, another client that contacted Dogtag while that certificate is in use could log it to a Certificate Transparency log (not a public one, unless your Dogtag CA is chained to a publicly trusted CA). If this occurred, there would be a record that the CA had misbehaved. What are the ramifications? If this happened in the public PKI, the offending CA would at best get a harsh and very public admonishment, and be put on notice. But trust store vendors might just straight up wash their hands of you and yank trust. In a private PKI is it such a big deal? Given our use case—the same subject names are used—probably not. But I leave it as an open topic to ponder how this might backfire. Conclusion In this post I introduced the pki-server cert-fix subcommand. The purpose of this tool is to simplify and speed up recovery when Dogtag system certificates have expired. It does what it says on the tin, with a few rough edges and, right now, a lot of caveats. The fundamentals are very good, but I think we need to address number of these caveats for cert-fix to be generally useful, especially in a FreeIPA context. Based on my early experiences and investigation, my suggested priorities are: Workaround for when the agent certificate is expired. This can affect every kind of deployment and the reliance on a valid agent certificate is a significant limitation. Workaround for expired subsystem certificate when TLS client authentication is used to bind to LDAP. This affects all FreeIPA deployments (standalone Dogtag deployments less commonly). Support renewing the CA certificate in cert-fix. A degree of sanity checking or confirmation may be reasonable (e.g. it must be explicitly listed on the CLI as a --cert option). Investigate ways to handle expired LDAP certificate, if issued by Dogtag. In some deployments, including some FreeIPA deployments, the LDAP certificate is not issued by Dogtag, so the risk is not universal. In writing this post I by no means wish to diminish Dinesh’s work. On the contrary, I’m impressed with what the tool already can do! And, mea culpa, I have taken far too long to test this tool and evaluate it in a FreeIPA setting. Now that I have a clearer picture, I see that I will be very busy making the tool more capable and ready for action in FreeIPA scenarios.