Upgrade Podman-based installations (Podman Quadlets)

This article describes how to upgrade an existing Podman Quadlets Installation of Smart ID Certificate Manager (CM).

Prerequisites

• Quadlet installation of Certificate Manager (CM) installed with version CM 8.10.x.

• Podman version 4.9.4 or later.

• Administrator's Workbench, Registration Authority, and Certificate Controller clients from CM distributable package.

• Access to the new version's distributable package including updated container images ,database migration scripts and configuration files.

Step-by-step instruction

1. Pre-upgrade preparations

1. Ensure database accessibility
   Verify that the database is accessible and functioning properly before proceeding:

   podman exec -ti <cf-server-container-name> bash
   # Use JDBC or other database tools to test the connection
   

2. Stop all running containers
   Use the following commands to stop the running containers:

   systemctl --user stop cf-server
   systemctl --user stop pgw
   

3. Backup the database.
   As part of the backup, make sure to backup the cmdb database, please refer to the relevant database documentation for this step.

4. Verify that the container is stopped
   Check that all containers are stopped:

   podman ps --all
   

   Ensure no containers from the previous deployment are running.

5. Backup Volumes (optional but highly recommended)
   If something goes wrong, the backup can be restored, and the system can return to its previous state.
   Volumes that should be backed up are certs, config, bin and any relevant volume to your installation.

   1. Locate the volumes using:

      podman volume list
      

      Example (name can change but should contain (cf-server):

      > podman volume list
      local       systemd-cf-server-certs
      local       systemd-cf-server-config
      local       systemd-cf-server-bin
      local       systemd-pgw-config-tomcat
      local       systemd-pgw-config-gw
      

   2. Locate the Mountpoint on host:

      podman volume inspect <volume-name>
      

      Example:

      > podman volume inspect systemd-cf-server-certs
      [
           {
                "Name": "systemd-cf-server-certs",
                "Driver": "local",
                "Mountpoint": "/home/azureuser/.local/share/containers/storage/volumes/systemd-cf-server-certs/_data",
                "CreatedAt": "2024-06-20T07:25:01.386573664Z",
                "Labels": {},
                "Scope": "local",
                "Options": {},
                "MountCount": 0,
                "LockNumber": 4
           }
      ]
      

   3. Back up the Mountpoint.

6. Unzip the distribution.

2. Load new container images

1. Import new images
   Load the new container images using:

   podman image load --input certificate-manager-<version>/images/cf-server-image-<version>.tar
   podman image load --input certificate-manager-<version>/images/pgw-image-<version>.tar
   

2. Verify the images
   Ensure the new images are loaded correctly:

   podman images
   

   Confirm the tags and versions of the new image.

3. Update Quadlet configuration

1. Locate the cf-server.container and pgw.container used for the previous installation.
   The following command displays the path of the files .container under the Loaded attribute.

    systemctl --user status cf-server
    systemctl --user status pgw
   

   Example

   > systemctl --user status cf-server
   × cf-server.service
        Loaded: loaded (/home/azureuser/.config/containers/systemd/cf-server.container; generated)
   ...
   

2. Modify configuration
   Modify the Image Name and tag to the new Image in the cf-server.container and pgw.container files.

3. Reload systemd daemon
   Reload the systemd daemon to apply the changes:

   systemctl --user daemon-reload
   

4. Upgrade database and configuration files

1. Run database scripts (only necessary for cf-server)
   The upgrade scripts are stored in the upgrade directory under the distributable.
   Execute the database upgrade script towards the database server.

   Ensure the script ran without errors.
   Examplefor postgresql:

   psql -U lcmreq -d cmdb -f "CMDBUpgrade_PostgreSQL_8_11_0.sql"
   

2. Update the configuration files
   Locate the .conf files under the upgrade folder inside the release distribution.

   Open the .conf files and update the configuration files according to the instructions specified by the .conf file. Upgrade should be done towards the volume Mountpoint.
   If the configuration path specified in the .conf file do not map to a volume, changes will automatically be applied hence these files can be ignored.

5. Upgrade Protocol Gateway

Version specific actions for upgrading Protocol Gateway can be found in step 3 of the Upgrade Protocol Gateway page .
Only update configuration files.

6. Restart the containers

1. Start the updated containers
   Start the new containers using:

   systemctl --user start cf-server
   systemctl --user start pgw
   

   Following the logs from startup can be done as follows:

   systemctl --user start cf-server;podman logs -f cf-server
   

2. Monitor logs
   Check the logs to ensure the containers started successfully:

   podman logs -f cf-server
   podman logs -f pgw
   

7. Post-upgrade validation

1. Verify connectivity
   Test connectivity using Administrator's Workbench and/or Registration Authority clients.

2. Verify configuration
   Access the container volumes to confirm that the configuration files are correctly applied:

   podman exec -ti cf-server bash
   ls /opt/cm/server/
   

3. Check database connection
   Confirm that the container is successfully connected to the upgraded database.

4. Health checks
   Enable and test the health check for the Protocol Gateway (pgw):

   systemctl --user start pgw
   podman logs -f pgw
   

8. Rollback procedure (if needed)

If any issue occurs during the upgrade, you can roll back using the following steps:

1. Stop the new containers

   systemctl --user stop cf-server
   systemctl --user stop pgw
   

2. Restore volumes
   Restore the backed-up volumes.

3. Point to the previous images
   Point to the old images in cf-server.container and pgw.container.

4. Restore database

5. Start the old containers
   Restart the containers using the old quadlets:

   systemctl --user daemon-reload
   systemctl --user start cf-server
   systemctl --user start pgw
   

9. Troubleshooting

View logs

Use the following command to view container logs for debugging:

podman logs <container-name>

Inspect container configuration

Inspect the container configuration for issues:

podman inspect <container-name>

Access shell in container

Start a shell inside the container for manual troubleshooting:

podman exec -ti cf-server bash

Check systemd status

Check the status of the systemd service:

systemctl --user status cf-server