Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Note
titleUnder update

This article is currently under update and will be finished within the coming weeks.

SEO Metadata
titleSet up high availability for Digital Access component

Smart ID Digital Access component supports distributed mode to enable high availability and failover that provides powerful flexibility and scaleabilityscalability. With this mode, Digital Access component will switch to a redundant service once the primary one has stopped working.

Smart ID Digital Access component supports distributed mode to enable high availability and failover that provides powerful flexibility and scaleabilityscalability. With this mode, Digital Access component will switch to a redundant service once the primary one has stopped working. Thereby, not only one but several redundant services are supported. Using high availability enables systems to meet high service-level agreement (SLA) requirements.

This article describes the setup of high availability between four Digital Access component instances. See for two Digital Access components with docker swarm and running services. See also High availability architecture for Digital Access component.

Note
  • Manager node is the node that hosts the administration service.
  • Worker node is a node that hosts other services, not running the administration service.

Expandall

Prerequisites

Expand
titlePrerequisites

The following prerequisites apply:

  • External databases are available for users, OATH and Oauth2.

Four instances of Digital Access component are running. in the examples in this article they are called HAG-1, HAG-2, HAG-3, and HAG-4:

Two instances shall be running in a protected networkand not be accessible from DMZ. (In the example in this article this is HAG-1 and HAG-2.)
  • Only one Administration service is installed. This is done on HAG-1.
  • Only one Distribution service is installed. This is done on HAG-1. 
  • Two instances shall be hosted in DMZ and run only Access Point. (In the example in this article this is HAG-3 and HAG-4.)

  • The Access Points running in DMZ should have access to the protected network so that they can communicate with other services like Authentication, Policy and Distribution service. For information how to set this up, refer to Run Digital Access component in distributed mode

  • The IP addresses of all four nodes must be known.

    Info

    When using the virtual appliance for deployment, the IP address for a node can be found in the console menu, select 1) Setup system > 1) modify interfaces. Here you can see the assigned IP to eth0.

    When using the Orchestrator deployment, use the IP addresses from the host network. 

    Step-by-step instruction

    Configure database service

    Expand
    titleConfigure database service

    To be able to add an Authentication service you must first point the user database, the OATH database, and the OAuth 2.0 database to an external database.

    Change hosts and add services

    • The following ports shall be open to traffic to and from each Docker host participating on an overlay network:
      • TCP port 2377 for cluster management communications
      • TCP and UDP port 7946 for communication among nodes
      • UDP port 4789 for overlay network traffic
    • For more details refer to: https://docs.docker.com/network/overlay/
    • Keep a note of IP addresses of nodes where access point is running.

    Step-by-step instruction

    Get token and stop services - manager node

    Expand
    titleGet cluster join token
    1. SSH to the node running administration service, that is, the manager node.
    2. Get the cluster join token by running this command. This token will be used for joining worker nodes to the manager node.

      Code Block
      titleGet token
      sudo docker swarm join-token worker

      Output of the command will be like:

      Panel
      titleOutput

      docker swarm join --token SWMTKN-1-5dxny21y4oslz87lqjzz4wj2wejy6vicjtqwq33mvqqni42ki2-1gvl9xiqcrlxuxoafesxampwq 192.168.253.139:2377



    Expand
    titleStop services
    1. Stop the

    services of all instances except the Administration service of HAG-1.To stop a service:
    1. running services.

      Code Block
      titleStop services
      sudo docker stack rm <your da stack name>


    Join as worker nodes

    Issue this command to stop, for example, the authentication service:

    Excerpt

    Do these steps on all worker nodes.

    Issue this command to stop, for example, the authentication service:

    Expand
    titleUsing virtual appliance
    Code Block
    sudo /etc/init.d/authentication-service stop
    Expand
    titleUsing Orchestrator
    Code Block
    docker exec orchestrator hagcli -s authentication-service -o stop
    Expand
    titleLog in to Digital Access Admin
    1. Log in to Digital Access Admin of HAG-1 with an administrator account.
    Expand
    titleChange hosts and add services
    1. Change the host of registered Policy service:
      1. In Digital Access Admin of HAG-1, go to Manage System > Policy Services.
      2. Select the registered Policy Service.
      3. Change the Internal Host from 127.0.0.1 to HAG-1's IP Address.
      4. Check Distribute key files automatically.
      5. Click Save.
    2. Add a Policy Service:
      1. In Digital Access Admin of HAG-1, go to Manage System > Policy Services.
      2. Click Add Policy Service…
      3. In Display Name enter "Policy Service 2".
      4. In Internal Host enter the IP address of HAG-2.
      5. Check Distribute key files automatically.
      6. Select the Server Certificate and Add it.
      7. Note down the Service ID of newly added Policy Service.
    3. Change the host of registered Distribution Service:
      1. In Digital Access Admin of HAG-1, go to Manage System > Distribution Services.
      2. Select the registered Distribution Service.
      3. Change the Internal Host from 127.0.0.1 to HAG-1's IP Address.
      4. Check Distribute key files automatically.
      5. Click Save.
    4. Change the host of registered Authentication Service:
      1. In Digital Access Admin of HAG-1, go to Manage System > Authentication Services.
      2. Select the registered Authentication Service.
      3. Change the Internal Host from 127.0.0.1 to HAG-1's IP Address.
      4. Check Distribute key files automatically.
      5. Click Save.
    5. Add an Authentication Service:
      1. In Digital Access Admin of HAG-1, go to Manage System > Authentication Services.
      2. Click Add Authentication Service…
      3. In Display Name enter "Authentication Service 2".
      4. In Internal Host enter the IP address of HAG-2.
      5. Check Distribute key files automatically.
      6. Select the Server Certificate and Add it.
      7. Note down the Service ID of newly added Authentication Service.
    6. Change the host of registered Access Point:
      1. In Digital Access Admin of HAG-1, go to Manage System > Access Points.
      2. Select the registered Access Point.
      3. Change the Internal Host to the IP Address of HAG-3. Make sure you use the correct IP address of HAG-3 (which is accessible in the protected network).
      4. Check Distribute key files automatically.
      5. Click Save.
    7. Add an Access Point:
      1. In Digital Access Admin of HAG-1, go to Manage System > Access Points.
      2. Click Add Access Point…
      3. In Display Name enter "Access Point 2".
      4. In Internal Host enter the IP address of HAG-4.
      5. Check Distribute key files automatically.
      6. Select the Server Certificate and Add it.
      7. Note down the Service ID of newly added Access Point.

    Set up services

    There will be only one administration service running, so all other Digital Access component instances need to have administration service at disabled state. You need to change the IP address of administration service in each Digital Access component instance in order to run the other services at external IP.

    Expand
    titleSet up HAG-1

    In the Administration console of HAG-1:

    Expand
    titleUsing virtual appliance
    1. Select 2) Detailed server setup.
    2. For each service:
      1. Select the service and answer "y" to the question "Should this service be enabled?"
      2. The question At which IP can I find [service] will be shown. Change the IP address of the service from 127.0.0.1 to the IP address of HAG-1.
    3. For the Access Point:
      1. Use the menu option 5) Setup Access Point and Disable the Access point, that is, answer "n" to the question "Should this service be enabled?"
    Expand
    titleUsing Orchestrator

    Enable all services

    Code Block
    docker exec orchestrator hagcli -s all -o enable
    Note

    The IP addresses under where each service is reachable has already been changed in the previous section.

    Expand
    titleSet up HAG-2

    In the Administration console of HAG-2:

    Expand
    titleUsing virtual appliance
  • Select 2) Detailed server setup.
  • For the Administration Service:
    1. Select the service and answer "n" to the question "Should this service be enabled?"
    2. The question At which IP can I find [service] will be shown. Change the IP address of the service from 127.0.0.1 to the IP address of HAG-1.
  • For the Access Point:
    1. Use the menu option 5) Setup Access Point and Disable the Access Point, that is, answer "n" to the question "Should this service be enabled?"
  • For each of the other services:
  • Select the service and answer "y" to the question "Should this service be enabled?"
  • The question At which IP can I find [service] will be shown. Change the IP address of the service from 127.0.0.1 to the IP address of HAG-2.
  • The questionWhat node Id does this service have? will be shown. Change the ID to the Service ID which you noted down in earlier steps corresponding to the service. Expand
    titleUsing Orchestrator

    For the Administration service:

    Disable the service:

    Code Block
    docker exec orchestrator hagcli -s administration-service -o disable
  • Change IP address of Administration Service for each service enabled on this host
    1. Open LocalConfiguration.xml in opt/nexus/primary/<service>/config/LocalConfiguration.xml 
    2. Search for Administration Service section
    3. Change value of mHost to external IP address of Administration Service.
  • For the Access Point:

    Disable the service:

    Code Block
    docker exec orchestrator hagcli -s access-point -o disable
    For each of the other services:

    Enable the service: 

    Code Block
    docker exec orchestrator hagcli -s <service> -o enable
    Expand
    titleSet up HAG-3 and HAG-4

    In the Administration console of HAG-3 and HAG-4:

    Expand
    titleUsing virtual appliance
    1. Select 2) Detailed server setup.
      1. For the Administration Service:
        1. Select the service and answer "n" to the question "Should this service be enabled?"
        2. The question At which IP can I find [service] will be shown. Change the IP address of the service from 127.0.0.1 to the IP address of HAG-1.
      2. For the Access Point:
        1. Use the menu option 5) Setup Access Point and answer "y" to the question "Should this service be enabled?"
        2. The question What node ID does the service have? will be shown. Change the node ID to the Service ID which you noted down in earlier steps.
      3. For each of the other services:
        1. Select the service and answer "n" to the question "Should this service be enabled?"
    Expand
    titleUsing Orchestrator

    For the Administration service:

    Disable the service:

    Code Block
    docker exec orchestrator hagcli -s administration-service -o disable
  • Change IP address of Administration Service for the Access point:
    1. Open LocalConfiguration.xml in opt/nexus/primary/access-point/config/LocalConfiguration.xml 
    2. Search for Administration Service section
    3. Change value of mHost to external IP address of Administration Service.
  • Change Service ID of Access point:
    1. Open LocalConfiguration.xml in opt/nexus/primary/access-point/config/LocalConfiguration.xml 
    2. Change id values in element <id> and attribute mId to a the number you got when adding the new service node in the administration interface
  • For the Access Point:

    Enable the service:

    Code Block
    docker exec orchestrator hagcli -s access-point -o enable
    For each of the other services:

    Disable the service: 

    Code Block
    docker exec orchestrator hagcli -s <service> -o disable
    Expand
    titleChange IP address of API resource

    Change the IP address of api resource:

    1. In Digital Access Admin, go to Manage Resource Access and click the +-sign at api.
      1. Click Edit Resource Host...
      2. Change the Host 127.0.0.1 to HAG-3 IP and HAG-4 IP. Separate the IP addresses with a semicolon (;).

    Set up load balancer

    Expand
    titleSet up load balancer

    To set up high availability for Digital Access component, an external load balancer must be used. In this example, we use HAProxy.

    1. Log in to Digital Access Admin of HAG-1 with an administrator account.
    2. In Digital Access Admin, go to Manage System > Access Points.
    3. For each added access point:
      1. Add a listener by clicking Add Additional Listener…
      2. In Host, enter the IP address of the Access Point. Enter a Port, and set Type to Load Balance.
      3. Click Add.
    4. Go to Manage System > Access Points.
    5. Select Configure Load Balancing…
    6. Check Enable multi-host sessions and Send sticky cookies. Enter a Name of Sticky Cookie to be used by the load balancer service.
    7. Click Save.
    8. Select Configure Load Balancing…
    9. Click Add Pair of Mirrored Access Points...
    10. Select Access Point 1 and Access Point 2 as Primary and Secondary server.
    11. Click Save.

    Check setup

    Expand
    titleCheck the setup
    1. Start all the required services.
    2. Publish the configuration.
    3. Check that all services are connected.
    4. Do a login to the portal and check if all works as expected and that you can see the portal items and display names properly.
    5. In case of any failure, check if sha1sum of shared.key and internal.key for all connected services are the same. The keys can be found under /opt/nexus/<service>/keys/. 
    6. Inspect logs and address any unexpected errors.
    This article is valid from Digital Access 6.0
    Join the nodes as worker nodes
    1. SSH to the worker node(s).

    2. Stop the running services.

      Code Block
      titleStop services
      sudo docker stack rm <your da stack name>


    3. Get the node ID.

      Code Block
      titleGet node ID
      sudo docker node ls


    4. Remove the labels.

      Code Block
      titleRemove labels
      sudo docker node update --label-rm  da-accesspoint <nodeid>
      sudo docker node update --label-rm  da-authentication <nodeid>
      sudo docker node update --label-rm  da-distribution <nodeid>
      sudo docker node update --label-rm  da-policy <nodeid>
      sudo docker node update --label-rm  da-admin <nodeid>


    5. if you are using PostgreSQL as database then remove label using this command (not to run on PostgreSQL node):

      Code Block
      titleIf using PostgreSQL
      sudo docker node update --label-rm  postgres <nodeid>


    6. Remove the node from the current swarm.

      Code Block
      titleRemove node
      sudo docker swarm leave --force


    7. Join to manager swarm using the command output from "Get cluster join token" above.

      Code Block
      titleExample of output of 'get token' command
      docker swarm join --token SWMTKN-1-5dxny21y4oslz87lqjzz4wj2wejy6vicjtqwq33mvqqni42ki2-1gvl9xiqcrlxuxoafesxampwq 192.168.253.139:2377
      


    8. On success, the output will be: This node joined a swarm as a worker.


    Remove labels at manager node

    Expand
    titleRemove labels at manager node
    1. SSH to manager node.
    2. Remove label for all services which are not required on this node.

      Code Block
      titleRemove label
      sudo docker node update --label-rm  da-accesspoint <nodeid>


    Edit configuration files

    Expand
    titleEdit configuration files

    Navigate to the docker-compose folder and edit these files:

    • docker-compose.yml
    • network.yml
    • versiontag.yml

    docker-compose.yml

    For each service, add one section in the docker-compose.yml file.

    For example, if you want to deploy two policy services on two nodes you will have two configuration blocks as shown in the example below.  

    Example:

    Change the values for the following keys:

    • Service name
    • Hostname
    • Constraints
    No Format
    policy1: 
    
    image:nexusimages.azurecr.io/smartid-digitalaccess/policy-service:6.0.5.60259 
        hostname: policy1 
        deploy: 
          mode: replicated 
          replicas: 1 
          placement: 
            constraints: 
             #If you need to set constraints using node name 
             #- node.hostname ==<node name>  
             # use node label 
             [node.labels.da-policy1 == true ] 
          resources: 
            limits: 
              cpus: "0.50" 
              memory: 512M 
            reservations: 
              cpus: "0.10" 
              memory: 128M 
        volumes: 
          - /opt/nexus/config/policy-service:/etc/nexus/policy-service:z 
          - /etc/localtime:/etc/localtime 
          - /etc/timezone:/etc/timezone 
        logging: 
          options: 
    max-size: 10m 
     
    policy2: 
    
        # configure image tag from versiontag.yaml 
        image:nexusimages.azurecr.io/smartid-digitalaccess/policy-service:6.0.5.60259 
        hostname: policy2 
        deploy: 
          mode: replicated 
          replicas: 1 
          placement: 
            constraints: 
             #If you need to set comnstraints using node name 
             #- node.hostname ==<node name>  
             # use node label 
             [node.labels.da-policy2 == true ] 
          resources: 
            limits: 
              cpus: "0.50" 
              memory: 512M 
            reservations: 
              cpus: "0.10" 
              memory: 128M 
        volumes: 
          - /opt/nexus/config/policy-service:/etc/nexus/policy-service:z 
          - /etc/localtime:/etc/localtime 
          - /etc/timezone:/etc/timezone 
        logging: 
          options: 
    max-size: 10m 
    
    

    network.yml

    For each service add network configuration in the network.yml file. For example, if you want to deploy two policy services on two nodes you will have two blocks of configuration as shown below.

    Example:

    Change the value of:

    • Service name: Service name should be identical to what is mentioned in docker-compose.yml


    No Format
    policy1: 
        ports: 
          - target: 4443 
            published: 4443 
            mode: host 
        networks: 
          - da-overlay 
     
           
      Policy2: 
        ports: 
          - target: 4443 
            published: 4443 

    Also make sure all the listeners that are used for access point Load balance are exposed on network.yml.

    versiontag.yml

    Add one line for each service in this file also.

    For example, if you have two policy services with name policy1 and policy2, you will have two lines for each service.

    Example:

    No Format
    Policy1: 
        image: nexusimages.azurecr.io/smartid-digitalaccess/policy-service:6.0.5.60259 
    policy2: 
        image: nexusimages.azurecr.io/smartid-digitalaccess/policy-service:6.0.5.60259 


    At manager node

    Expand
    titleVerify and identify nodes
    1. Verify if all nodes are part of cluster by running this command.

      Code Block
      titleVerify if all nodes are part of cluster
      sudo docker node ls

      Example:
      Image Added

    2. Identify nodes ID, master and worker where the service will be distributed.

      Code Block
      titleIdentify nodes
      sudo docker node inspect --format '{{ .Status }}' h9u7iiifi6sr85zyszu8xo54l


    3. Output from this command:

      No Format
      {ready  192.168.86.129}

      IP address will help to identify the Digital Access node


    Expand
    titleAdd new labels for each service

    Add new labels for each service which you want to run in worker nodes. In this example, we have used “2” as postfix for each service name. You can choose any name based on your requirement, but make sure they are in accordance with what we have defined in constraint section in the docker-compose.yml file.

    1. Use these commands to add label for each service:

      Code Block
      titleCommands to add labels
      sudo docker node update --label-add da-policy2=true <node ID> 
      sudo docker node update --label-add da-authentication2 =true <node ID> 
      sudo docker node update --label-add da-accesspoint2=true <node ID> 
      sudo docker node update --label-add da-distribution2=true <node ID>


    2. Deploy your stack using this command. To run the command your working directory should be docker-compose.

      Code Block
      titleDeploy DA stack
      sudo docker stack deploy --compose-file docker-compose.yml -c network.yml -c versiontag.yml <your da stack name>

      Here: 

      • docker stack deploy is the command to deploy services as stack. 
      • compose file flag is used to provide the file name of base docker-compose file. 
      • -c is short for –compose-file flag. It is used to provide override files for docker -compose. 
      • <your da stack name> is the name of the stack. You can change it based on requirements. 

    In Digital Access Admin

    Expand
    titleDo updates in Digital Access Admin
    1. Log in to Digital Access Admin and change the internal host and port for each added service according to the docker-compose.yml and network.yml files.
    2. Go to Manage System > Distribution Services and
      1. select the checkbox “Listen on all Interfaces” in case of the ports that are to be exposed
      2. also select the checkbox “Distribute key files automatically”.

        Image Added
    3. Go to Manage System >Access Points and provide the IP address instead of the service name. Also enable the "Listen on all Interfaces" option.

      Image Added

    Do final steps

    Expand
    titleDo final steps
    1. Make sure all services are stopped, else remove stack using this command.

      Code Block
      titleRemove stack
      sudo docker stack rm <da/stack Name> 


    2. In worker node, edit service Local Configuration file, and provide values for:

      <core>      <id>6</id>    </core>

      <attribute name="mHost" type="string" value="policy1"/>

      <attribute name="mId" type="integer" value="6"/>

    3. Copy the keys from manager node to worker node services.

      Note

      For access point: copy only shared key.

      For all services enabled in worker node: copy internal and shared keys.


      Code Block
      titleCopy keys
      /opt/nexus/config/administration-service/keys# scp internal.key agadmin@<worker node Ip>:/home/agadmin
      
      opt/nexus/config/administration-service/keys# scp shared.key agadmin@@<worker node Ip>:/home/agadmin


    4. Restart services using these commands.

      Code Block
      titleRestart services
      sudo docker stack deploy --compose-file docker-compose.yml -c network.yml -c versiontag.yml <your da stack nam> 


    5. For Database connection issue, enter this command.

      Code Block
      titleDatabase connection issue
      Restart postgres container  docker stop <postgres container ID> 

      Check connection in Digital Access Admin for IP and password provided.