Health Checker module

Health Checker is a utility that monitors the state of Docker containers and platform services. The utility helps find:

• failed containers and services;
• services that could not be matched to any container (orphans).

Health Checker:

• tracks the state of Docker containers — running, stopped, restarting;
• checks services inside containers through supervisorctl;
• compares container services with services registered in the Consul service discovery system;
• prints the collected data as a table in the terminal;
• runs from its own container and accesses Docker through a socket.

How it works

To get information about the system state, Health Checker:

1. Collects data about containers.
2. Analyzes information about services in containers.
3. Matches container information with data in Consul.

Collecting data about containers

Health Checker gets the list of containers from two sources:

• /opt/ispsystem/vm/config.json — base64_compose blocks are decoded from the Patches section;
• /opt/ispsystem/vm/docker-compose.yaml — the list of Docker Compose services is taken from the file.

Then, through the Docker API over a socket, Health Checker requests the current list of containers with full data: network settings (NetworkSettings), state (State), status (Status).

Analyzing services in containers

For each running container, Health Checker runs the supervisorctl status command through docker exec and parses the output to get the status of each service: RUNNING, STOPPED, FATAL, BACKOFF.

From the supervisor configuration (/opt/supervisor.d/*.conf), Health Checker extracts the --name, --port, and --mode parameters. These parameters are used to match a service with a service in Consul.

Matching services with Consul

Consul integration

Health Checker queries two Consul API endpoints:

• /v1/agent/services — list of registered services;
• /v1/health/state/any — health-check results for each service.

Based on this data, Health Checker matches container services with Consul services.

Registering expected identifiers

For each service that has the --name parameter set in the supervisor configuration, Health Checker builds two possible Consul service identifiers (ServiceID):

• by name and mode: {consul_service_name}_{mode}_{short_container_id};
• by IP address and port: {container_ip}_{port}.

Health Checker stores both variants in an internal dictionary for later lookup.

Synchronization with Consul

When data is received from /v1/agent/services, Health Checker looks for a match in the following order:

1. Direct match — the ServiceID from Consul is found among the identifiers created in the previous step. This is an exact match.
2. IP:Port match — if the ServiceID is not found, Health Checker compares the Address:Port pair from the Consul data with the container IP address and port.
3. Container orphans — a Consul service has the container IP address, but it is not matched to any service. Health Checker binds such a service to the container as a container orphan: in the table, it is shown in that container's row with the + prefix. If a matching service appears later, the service will be matched to it.
4. Full orphans — Consul services that could not be linked to any container. Health Checker shows them as a separate Consul orphans block at the bottom of the table.

Updating health-check statuses

Through /v1/health/state/any, for each matched service Health Checker updates:

• health-check status — passing, warning, critical, maintenance;
• command output (output) — for example, HTTP POST http://172.18.0.10:200/health: 200 OK.

Tracking changes

When a container state changes (transitions to RUNNING or out of it), Health Checker re-registers or removes its services from the matching system. Health Checker marks services that disappeared from Consul with the REMOVED status.

Installation and startup

Installation for version 2026.08.1 and later

To install Health Checker, in the right menu, click the icon → section Modules → module Health Checker → button Install .

To start Health Checker:

1. Connect to the server with the platform via SSH. For more information about SSH connection, see the article Workstation setup.

2. Run the command:

   docker exec -it health_check check <parameters>

   Explanations

   • health_check — the name of the Health Checker container (set by the container_name parameter in the patch file);
   • check — the command that starts the utility inside the container;
   • -it — docker exec flags for an interactive terminal. They are needed so that Health Checker colors the table and scrolling works in continuous monitoring mode (see the Continuous monitoring section);
   • <parameters> — launch parameters.

Installation for version 2026.06.1 and below

To install Health Checker, add it to the platform as a patch:

1. Connect to the server with the platform via SSH. For more information about SSH connection, see the article Workstation setup.

2. Create the patch file health-checker.yaml:

   services:
     health_check:
       image: docker-registry.ispsystem.com/team/vm/health-checker/debian-12:1.0.1
       container_name: health_check
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock
         - /opt/ispsystem/vm/docker-compose.yaml:/opt/ispsystem/vm/docker-compose.yaml:ro
         - /opt/ispsystem/vm/config.json:/opt/ispsystem/vm/config.json:ro
       environment:
         - LOG_LEVEL=DEBUG
         - DOCKER_HOST=unix:///var/run/docker.sock
         - HEALTH_CHECKER_CONFIG_PATH=/opt/ispsystem/vm/config.json
         - HEALTH_CHECKER_COMPOSE_PATH=/opt/ispsystem/vm/docker-compose.yaml
       networks:
         - vm_vm_box_net
       stdin_open: true
       tty: true
       command: tail -f /dev/null
   
   networks:
     vm_vm_box_net:
       external: true

   Explanations

   • volumes — gives the Health Checker container access to the platform Docker socket, as well as to the docker-compose.yaml and config.json files in read-only mode (:ro);
   • DOCKER_HOST — stores the address of the Docker socket through which Health Checker accesses the Docker API;
   • HEALTH_CHECKER_CONFIG_PATH and HEALTH_CHECKER_COMPOSE_PATH — stores paths to the platform files that Health Checker reads to get the list of expected containers (see the Collecting data about containers section);
   • command: tail -f /dev/null — keeps the container running so that commands can be executed inside it with docker exec.

3. Add the patch to the platform:

   vm add-patch -p health -f health-checker.yaml

   Explanations

   • -p health — patch name;
   • -f health-checker.yaml — path to the patch file.

4. Start Health Checker:

   docker exec -it health_check check <parameters>

   Explanations

   • health_check — the name of the Health Checker container (set by the container_name parameter in the patch file);
   • check — the command that starts the utility inside the container;
   • -it — docker exec flags for an interactive terminal: they are needed so that Health Checker colors the table and scrolling works in continuous monitoring mode (see the Continuous monitoring section);
   • <parameters> — launch parameters.

Launch parameters

The --format json parameter outputs the same data as the table in a structured form: a summary (summary) with counters and a list of problems, a list of containers with services (containers), and a list of full orphans (orphans). The json format allows you to send the check result to external monitoring systems.

{
  "check_id": "f5485278",
  "summary": {
    "has_problems": false,
    "problems_count": 0,
    "total_containers": 1,
    "running_containers": 1,
    "stopped_containers": 0,
    "total_services": 3,
    "healthy_services": 3,
    "unhealthy_services": 0,
    "total_orphans": 2,
    "problems": []
  },
  "containers": [
    {
      "name": "auth",
      "short_id": "fb6151ceed1d",
      "state": "running",
      "status": "Up 3 days",
      "ip": "172.18.0.10",
      "services": [
        {
          "name": "auth_reader",
          "status": "running",
          "config": {
            "consul_service_name": "auth_v4",
            "mode": "reader",
            "port": 200,
            "expected_in_consul": true
          },
          "consul": {
            "consul_status": "passing",
            "consul_id": "auth_v4_reader_fb6151ceed1d"
          }
        }
      ],
      "container_orphans": []
    }
  ],
  "orphans": []
}

JSON response notes:

• summary.problems — the list of problematic services and containers found in a single check run;
• containers[].services — supervisor services inside a container; with --full, includes services in the RUNNING status;
• containers[].container_orphans — partial orphans bound to this container by IP address (see the Synchronization with Consul section);
• orphans — full Consul orphans not bound to any container.

Display modes

Table structure

Health Checker outputs a table with columns:

For example, for the vm_box container, Health Checker in full mode outputs:

┼─────────────────────┼─────────┼─────────────────────┼─────────────────────┼
│ Name                │ State   │ Details             │ Consul              │
┼─────────────────────┼─────────┼─────────────────────┼─────────────────────┼
│ vm_box              │ running │ Up 3 days           │                     │
│                     │         │ 172.18.0.13         │                     │
│ ├── checker         │ running │ pid 166  uptime 3   │ Status: passing,    │
│                     │         │ days Port: 2400     │ Name: checker_v3(…) │
│ ├──                 │ exited  │ Jul 28 02:40 AM     │  -                  │
│ rdns_auto_enabler   │         │                     │                     │
│ + gosockify         │ -       │ -                   │ Status: passing,    │
│                     │         │                     │ Name: gosockify(…)  │
┼─────────────────────┼─────────┼─────────────────────┼─────────────────────┼

In this example, checker and rdns_auto_enabler are supervisor services inside the vm_box container (the second one is stopped), and gosockify is a container orphan: a Consul service with the IP address of the vm_box container that Health Checker could not match to any supervisor service.

Color coding

Health Checker colors table rows by status:

• green — everything is OK (RUNNING, passing);
• yellow — warnings (WARNING, BACKOFF, not checked, partially working container services);
• red — problems (FATAL, CRITICAL, exited, NOT_FOUND for expected services).

View modes

By default, Health Checker shows a shortened output. You can change the output verbosity with launch parameters.

--full (full mode)

The --full parameter shows all services in all containers with full Consul identifiers, including services in the RUNNING status with a passing health-check. For example, for the auth container, the check --full --containers auth command outputs all three supervisor services with their ports and Consul identifiers, whereas without --full Health Checker shows only the services: 3/3 counter.

Short Mode (default)

Without the --full parameter, Health Checker shows only:

• a list of containers with the number of healthy services, for example services: 6/6;
• services in a status other than RUNNING;
• services with failing health-checks in Consul.

--problems

The --problems parameter is a stricter filter: it shows only containers that have at least one problem. 

--no-orphans

The --no-orphans parameter hides the Consul orphans block with full orphans. Use this parameter if full orphans are not related to the monitored services and are not needed for the current check.

Continuous monitoring

The --watch parameter switches Health Checker to continuous monitoring of the system state. 

Watch mode behavior

In --watch mode, Health Checker:

• requests data every two seconds;
• collects data in a background thread, so you can scroll the table with the mouse while it is refreshing;
• when the container state changes, updates the entire data chain associated with it;
• when a container restarts, automatically rematches its services with Consul.

What is tracked in real time

In --watch mode, Health Checker tracks:

• the appearance and disappearance of containers. If a container disappears from the list of running containers, Health Checker marks it with the UNKNOWN status;
• changes in container state — start, stop, restart;
• changes in supervisor service status, for example, RUNNING → STOPPED → FATAL;
• changes in Consul health-check status, for example, PASSING → WARNING → CRITICAL;
• loss of registration in Consul — the service was registered, but disappeared from /v1/agent/services.

Output control

You can move around the table with the mouse wheel or keys:

• j — line down;
• k — line up;
• d — page down;
• u — page up;
• g — to the beginning of the table;
• G — to the end of the table.

To exit the utility, press q.