Controller account config has been lost after service restarted on error

[avonurt]

What has happened?

The logs are attached below. On the records, you could see that service has got an Error response from external service https://sqtn.api.onfinality.io/public and restarted. After the service resumed, it continuously reported WARN no valid controller account config in service.

How do we solve the issue?

1. We restarted the service docker-compose restart on the first step. It did not solve the issue.
2. In the second step, we reconfigured the controller account from the admin page, which solved the issue.

Logs

2022-04-28T02:46:13.779Z <transaction> INFO report project status: SUCCEED 
2022-04-28T02:49:11.160Z <transaction> INFO report project status: PROCESSING | project QmTLPAeVhNNxU69NhcsKHRrCXBhmQYykqkszwETRS75A76 | time: 1651114128 | block height: 10066359 
2022-04-28T02:49:56.155Z <transaction> INFO report project status: SUCCEED 
2022-04-28T02:49:58.776Z <transaction> INFO report project status: PROCESSING | project QmSzPQ9f4U1GERvN1AxJ28xq9B5s4M72CPvotmmv1N2bN7 | time: 1651114182 | block height: 9662480 
/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:233
        var error = new Error(message);
                    ^

Error: bad response (status=502, headers={"content-type":"text/plain; charset=utf-8","content-length":"93","connection":"close","date":"Thu, 28 Apr 2022 02:50:32 GMT","x-kong-upstream-latency":"5592","x-kong-proxy-latency":"0","via":"kong/2.1.4"}, body="i/o timeout. The gateway cannot get a response, please try again or contact the administrator", requestBody="{\"method\":\"eth_getTransactionByHash\",\"params\":[\"0x312c4a5052a3e603f83ab76ac7f6678f2e08fd24e247aaffef37bbf51f3a1a50\"],\"id\":31496,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="https://sqtn.api.onfinality.io/public", code=SERVER_ERROR, version=web/5.6.0)
    at Logger.makeError (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:233:21)
    at Logger.throwError (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/logger/lib/index.js:242:20)
    at /usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:267:36
    at step (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:33:23)
    at Object.next (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:14:53)
    at fulfilled (/usr/local/lib/node_modules/@subql/indexer-coordinator/node_modules/@ethersproject/web/lib/index.js:5:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  reason: 'bad response',
  code: 'SERVER_ERROR',
  status: 502,
  headers: {
    'content-type': 'text/plain; charset=utf-8',
    'content-length': '93',
    connection: 'close',
    date: 'Thu, 28 Apr 2022 02:50:32 GMT',
    'x-kong-upstream-latency': '5592',
    'x-kong-proxy-latency': '0',
    via: 'kong/2.1.4'
  },
  body: 'i/o timeout. The gateway cannot get a response, please try again or contact the administrator',
  requestBody: '{"method":"eth_getTransactionByHash","params":["0x312c4a5052a3e603f83ab76ac7f6678f2e08fd24e247aaffef37bbf51f3a1a50"],"id":31496,"jsonrpc":"2.0"}',
  requestMethod: 'POST',
  url: 'https://sqtn.api.onfinality.io/public'
}
2022-04-28T02:50:33.960Z <nestjs> INFO Starting Nest application... 
2022-04-28T02:50:33.992Z <nestjs> INFO TypeOrmModule dependencies initialized 
2022-04-28T02:50:33.992Z <nestjs> INFO ConfigureModule dependencies initialized 
2022-04-28T02:50:33.992Z <nestjs> INFO AppModule dependencies initialized 
2022-04-28T02:50:33.993Z <nestjs> INFO ServeStaticModule dependencies initialized 
2022-04-28T02:50:33.993Z <nestjs> INFO GraphQLSchemaBuilderModule dependencies initialized 
2022-04-28T02:50:33.993Z <nestjs> INFO GraphQLModule dependencies initialized 
2022-04-28T02:50:34.034Z <nestjs> INFO TypeOrmCoreModule dependencies initialized 
2022-04-28T02:50:34.034Z <nestjs> INFO TypeOrmModule dependencies initialized 
2022-04-28T02:50:34.035Z <nestjs> INFO TypeOrmModule dependencies initialized 
2022-04-28T02:50:34.036Z <nestjs> INFO AccountModule dependencies initialized 
2022-04-28T02:50:34.038Z <nestjs> INFO ProjectModule dependencies initialized 
2022-04-28T02:50:34.052Z <nestjs> INFO AdminController {/}: 
2022-04-28T02:50:34.053Z <nestjs> INFO Mapped {/env.js, GET} route 
2022-04-28T02:50:34.149Z <nestjs> INFO Nest application successfully started 
2022-04-28T02:50:34.152Z <indexer-coordinator> INFO coordinator service started 
2022-04-28T02:50:34.152Z <indexer-admin> INFO indexer admin app started 
2022-04-28T02:50:34.163Z <contract> WARN no valid controller account config in service 
2022-04-28T02:50:34.164Z <network> INFO transaction interval: 300000 
2022-04-28T02:50:34.165Z <contract> WARN no valid controller account config in service 
2022-04-28T02:55:34.178Z <contract> WARN no valid controller account config in service 
2022-04-28T02:55:34.181Z <contract> WARN no valid controller account config in service 
2022-04-28T03:00:34.181Z <contract> WARN no valid controller account config in service 
2022-04-28T03:00:34.183Z <contract> WARN no valid controller account config in service 
2022-04-28T03:05:34.182Z <contract> WARN no valid controller account config in service 
2022-04-28T03:05:34.185Z <contract> WARN no valid controller account config in service 
2022-04-28T03:10:34.186Z <contract> WARN no valid controller account config in service 
2022-04-28T03:10:34.188Z <contract> WARN no valid controller account config in service 
2022-04-28T03:15:34.189Z <contract> WARN no valid controller account config in service 
2022-04-28T03:15:34.191Z <contract> WARN no valid controller account config in service 
2022-04-28T03:20:34.191Z <contract> WARN no valid controller account config in service 
2022-04-28T03:20:34.193Z <contract> WARN no valid controller account config in service 
2022-04-28T03:25:34.197Z <contract> WARN no valid controller account config in service 
2022-04-28T03:25:34.199Z <contract> WARN no valid controller account config in service 
2022-04-28T03:30:34.201Z <contract> WARN no valid controller account config in service 

[ianhe8x]

Can you check the docker-compose file to see where the postgres db dir is mounted from? there was a version that it was mounted from /var/tmp, which will be removed by os at some point of time, and we updated it to put it in the same folder as docker-compose file.

[avonurt]

Hi, here is the docker-compose file that we use:

version: "3"

services:
  postgres:
    image: postgres:12-alpine
    container_name: coordinator_db
    ports:
      - 5432:5432
    volumes:
      - .data/postgres:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgres
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 5s
      timeout: 5s
      retries: 5

  coordinator-service:
    image: onfinality/subql-coordinator:v0.1.6
    container_name: coordinator_service
    ports:
      - 8000:8000
    depends_on:
      "postgres":
        condition: service_healthy
    restart: always
    environment:
      DOCKER_DB: coordinator_db
    volumes:
      - /var/tmp/composeFiles:/var/tmp/composeFiles
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      - --network=testnet
      - --ws-endpoint=https://sqtn.api.onfinality.io/public
      - --postgres-host=postgres
      - --postgres-database=postgres
      - --postgres-port=5432
      - --port=8000
    healthcheck:
      test:
        [
          "CMD",
          "curl",
          "-i",
          "-X POST",
          "http://coordinator-service:8000/graphql",
          "-H 'Content-Type: application/json'",
          "-d '{ \"query\": \"query { accountMetadata { network } }\" }'"
        ]
      interval: 5s
      timeout: 5s
      retries: 5
  proxy-server:
    image: onfinality/subql-indexer-proxy:v0.1.5
    container_name: coordinator_proxy
    ports:
      - 80:80
    depends_on:
      "coordinator-service":
        condition: service_healthy
    restart: always
    command:
      - --port=80
      - --host=0.0.0.0
      - --secret-key=qmbyd5hwac2jdcu
      - --service-url=http://coordinator-service:8000/graphql
networks:
  default:
    name: cooridnator-service