Troubleshooting after upgrade Digital Access to 6.0.5 and above

Problem: After upgrade to 6.0.5 or above, publishing one time does not connect the services.

Solution:

1. Retry publishing or restart services and then publish again.
   
   If publishing multiple times even does not connect the services:
2. Verify that for each service, the LocalConfiguration file (/opt/nexus/config/<service>/config/LocalConfiguration.xml) has the "Administration Service" object mHost attribute's value as "admin" instead of Host IP/127.0.0.1.
3. If not, follow these steps:

1. 1. Change mhost to "admin".

      Change mhost to "admin"

      CODE

      <node>
      <object key="c000ejp1m5" name="Administration Service" trans="ivjq0838gkxs" ver="50600">
      <attribute name="mAllInterfaces" type="boolean" value="false"/>
      <attribute name="mPort" type="integer" value="8300"/>
      <attribute name="mHost" type="string" value="admin"/>
      <attribute name="mType" type="integer" value="5"/>
      <attribute name="mId" type="integer" value="1"/>
      </object>

   2. Restart the Digital Access services using the following commands:
      

      Restart services

      CODE

      sudo docker stack rm da
      sudo bash /opt/nexus/scripts/start-all.sh

Problem: The access point does not connect and shows '403 Access Denied' when accessed.

Solution: Either restart all services or go to policy service setting in Digital Access Admin and click on Save and then Publish.

Problem: When restarting the services, if you get this message: 'da overlay network is missing' [Error response from daemon: network da_da-overlay not found]

Solution: Restart all the services again

Problem: Database max connection issue

Solution: Restart the postgres container.

Problem: If the nodes are not labelled correctly, the services may not start. The node labels determine which services are running on that node. For example, an Access Point not starting in the internal network, is most likely a label problem.

1. Enter the command docker stack ps da to find out if any service does not start on the indicated node.

Solution:

1. List the currently configured labels:
   

   List labels

   CODE

   docker node ls -q | xargs docker node inspect -f '{{ .ID }} [{{ .Description.Hostname }}]: {{ range $k, $v := .Spec.Labels }}{{ $k }}={{ $v }} {{end}}'

   Example output:

   CODE

   zhozhf8idvre7app5yyuh1bpm [ubuntu2004-1]: da-access-point=true da-administration-service=true da-authentication-service=true da-distribution-service=true da-policy-service=true postgres=true
   ab37r22u342pdp478994cf23 [ubuntu2004-2]: da-access-point1=true da-access-point=true 

   In the above example output, the da-access-point will try to start in both nodes, which is unlikely to be the intention.

2. 1. So to remove the da-access-point label from the example in step 1, enter this command:
      

      Example: Remove a label

      CODE

      docker node update --label-rm da-access-point ab37r22u342pdp478994cf23

      To remove a label use this command:
      

      Syntax: Remove a label

      CODE

      docker node update --label-rm <label> <node id>

Problem: When entering the command docker ps or docker stack ps da, some or all Digital Access services are missing, or appeare only a few seconds.

This is because there is some sort of problem on the startup of the services. The docker ps command quickly hide any stopped containers so they are either completely missing from docker ps or only there for a few seconds after which they fail. Since the desired state of the services is to be running, docker will try to start the containers over and over again, but if the problem is permanent it will keep failing.

Solution:

1. To get more information, enter this command:

   Show all containers, including stopped ones

   CODE

   docker ps -a
   

   This will produce a long list where the container which was last to stop is in the top.

2. Find the ID of that container (alphanumeric string) and enter this command:

   Show logs for container

   CODE

   docker logs <container-id>
   

   In most cases this will give a good hint of the problem. 

Docker is quick at cleaning up stopped containers. So a pause of more than a minute between the two commands will most likely result in a "container not found" error message.