Create NestJS health check library

Reading Time: 6 minutes

 91 total views,  2 views today

Introduction

NestJS applications in my company implement health checking for k8s liveness probe. These applications check the following items are up and running: backend, database, redis, bull queue and external servers. Eventually, different backend teams encapsulate similar endpoint and services in a heath check module and import it to the application. This approach is not DRY and I decided to create a heath check library and reuse it in applications to replace the custom health check module.

In this blog post, I show the codes to create a nestjs health check library and use it in a sample application to check the health of backend, Postgres database, redis, bull queues and doc website of nestjs and angular.

let's go

Create health check library

First, create a new project called nestjs-health. Rename src/ to lib/ and update tsconfig.json and package.json respectively.

// Directory tree
lib
├── controllers
│   ├── health.controller.ts
│   └── index.ts
├── health-module-definition.ts
├── health.module.ts
├── index.ts
├── indicators
│   ├── bull-queue.health.ts
│   └── index.ts
├── interfaces
│   ├── health-module-options.interface.ts
│   └── index.ts
├── services
│   ├── health.service.ts
│   └── index.ts
└── types
    ├── index.ts
    └── redis-connection-options.type.ts
// tsconfig.json
{
  "compilerOptions": {
    "module": "commonjs",
    "declaration": true,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "target": "es2017",
    "sourceMap": true,
    "outDir": "./dist",
    "rootDir": "./lib",
    "incremental": true,
    "skipLibCheck": true,
    "strictNullChecks": false,
    "noImplicitAny": false,
    "strictBindCallApply": false,
    "forceConsistentCasingInFileNames": false,
    "noFallthroughCasesInSwitch": false,
    "esModuleInterop": true,
    "strict": true,
    "strictPropertyInitialization": false,
    "noImplicitThis": true,
    "noUnusedParameters": true,
    "noUnusedLocals": true,
    "noUncheckedIndexedAccess": true
  },
  "include": ["lib/**/*"],
  "exclude": ["dist", "test", "node_modules"]
}

Update outDir to ./dist and rootDir to ./lib. Include ['lib/**/*'] and excludes ["dist", "test", "node_modules"] from compilation

Add "main": "./dist/index.js" to package.json

Install dependencies

npm i --save-exact --save-dev @nestjs/terminus
npm i --save-exact @nestjs/microservices @nestjs/axios @nestjs/bull bull @nestjs/typeorm ts-node

Then, move some “dependencies” to “peerDependencies”

"peerDependencies": {
    "@nestjs/common": "^9.0.0",
    "@nestjs/core": "^9.0.0",
    "@nestjs/microservices": "^9.1.5",
    "@nestjs/axios": "^0.1.0",
    "@nestjs/bull": "^0.6.1",
    "bull": "^4.10.1",
    "@nestjs/typeorm": "^9.0.1",
    "ts-node": "^10.0.0"
}

Define HealthModuleOptions interface

First, we define HealthModuleOptions interface that enables us to pass configurations from application to the library

// redis-connection-options.type.ts

import { RedisOptions } from '@nestjs/microservices';

export type RedisConnectOptions = RedisOptions['options'];
// health-module-options.interface.ts

import { HealthIndicatorFunction } from '@nestjs/terminus';
import { RedisConnectOptions } from '../types';

export interface HealthModuleOptions {
  app: string;
  backendUrl: string;
  shouldCheckDatabase?: boolean;
  queueNames?: string[];
  redisOptions?: RedisConnectOptions;
  indicatorFunctions?: HealthIndicatorFunction[];
}
  • app – key of the backend application
  • backendUrl: URL of the backend application, for example, http://localhost:3000
  • shouldCheckDatabase: determine whether or not ping the database that uses TypeORM
  • queueNames: Name array of bull queues
  • indicatorFunctions: custom indicator functions designed to check components’ health

Define Health Module Definition

Define a ConfigurableModuleBuilder to export ConfigurationModuleClass and MODULE_OPTIONS_TOKEN.

Invoke setClassMethodName to override the static methods of HealthModule to forRoot and forRootAsync

import { ConfigurableModuleBuilder } from '@nestjs/common';
import { HealthModuleOptions } from './interfaces';

export const { ConfigurableModuleClass, MODULE_OPTIONS_TOKEN } = new ConfigurableModuleBuilder<HealthModuleOptions>()
  .setClassMethodName('forRoot')
  .build();

Create module for the health check library

Create HealthModule that extends ConfigurableModuleClass

import { Module } from '@nestjs/common';
import { TerminusModule } from '@nestjs/terminus';
import { ConfigurableModuleClass } from './health-module-definition';

@Module({
  imports: [TerminusModule],
  controllers: [],
  providers: [],
  exports: [],
})
export class HealthModule extends ConfigurableModuleClass {}

controllers, providers and exports are empty arrays but we will update them after generating services and controller.

Generate custom health indicator

The following code perform health check on arbitrary number of bull queues. A queue is up and can accept jobs when it connects to redis and redis status is ready.

// indicators/bull-queue.health.ts

import { getQueueToken } from '@nestjs/bull';
import { Injectable } from '@nestjs/common';
import { ModuleRef } from '@nestjs/core';
import { HealthIndicator, HealthIndicatorResult, HealthCheckError } from '@nestjs/terminus';
import { Queue } from 'bull';

@Injectable()
export class BullQueueHealthIndicator extends HealthIndicator {
  constructor(private moduleRef: ModuleRef) {
    super();
  }

  async isHealthy(queues: string[]): Promise<HealthIndicatorResult> {
    const promiseResults = await this.checkQueuesReady(queues);
    const errorResults = this.filterErrors(promiseResults);

    if (errorResults.length) {
      throw new HealthCheckError('Bull queue failed', this.getStatus('bull', false, { errors: errorResults }));
    }

    return this.getStatus('bull', true);
  }

  private filterErrors(promiseResults: PromiseSettledResult<boolean>[]): string[] {
    const errorResults: string[] = [];
    for (const promiseResult of promiseResults) {
      if (promiseResult.status === 'rejected') {
        if (promiseResult.reason instanceof Error) {
          errorResults.push(promiseResult.reason.message);
        } else {
          errorResults.push(promiseResult.reason);
        }
      }
    }
    return errorResults;
  }

  private async checkQueuesReady(queues: string[]) {
    const promises = queues.map(async (name) => {
      const queueToken = this.moduleRef.get(getQueueToken(name), { strict: false });
      if ((queueToken as Queue).isReady) {
        const queue = await (queueToken as Queue).isReady();
        const isEveryClientReady = queue.clients.every((client) => client.status === 'ready');
        if (!isEveryClientReady) {
          throw new Error(`${name} - some redis clients are not ready`);
        }
        return true;
      }
      throw new Error(`${name} is not a bull queue`);
    });

    return Promise.allSettled(promises);
  }
}

Create health checking service

In the health service, I inject MODULE_OPTIONS_TOKEN injection token to obtain a reference to HealthModuleOptions. Then, I examine the options to determine the components to perform health check.

// services/health.service.ts

import { MODULE_OPTIONS_TOKEN } from '../health-module-definition';
import { HealthModuleOptions } from '../interfaces';
import { BullQueueHealthIndicator } from '../indicators';

@Injectable()
export class HealthService {
  constructor(
    @Inject(MODULE_OPTIONS_TOKEN) private options: HealthModuleOptions,
    private health: HealthCheckService,
    private http: HttpHealthIndicator,
    private db: TypeOrmHealthIndicator,
    private bull: BullQueueHealthIndicator,
    private microservice: MicroserviceHealthIndicator,
  ) {}

  check(): Promise<HealthCheckResult> {
    const indicatorFunctions: HealthIndicatorFunction[] = [];

    indicatorFunctions.push(() => this.http.pingCheck(this.options.app, this.options.backendUrl));

    if (this.options.shouldCheckDatabase) {
      indicatorFunctions.push(() => this.db.pingCheck('database'));
    }

    if (this.options.redisOptions) {
      indicatorFunctions.push(() =>
        this.microservice.pingCheck<RedisOptions>('redis', {
          transport: Transport.REDIS,
          options: this.options.redisOptions,
          timeout: 5000,
        }),
      );
    }

    if (this.options.queueNames?.length) {
      indicatorFunctions.push(() => this.bull.isHealthy(this.options.queueNames));
    }

    if (this.options.indicatorFunctions?.length) {
      indicatorFunctions.push(...this.options.indicatorFunctions);
    }

    return this.health.check(indicatorFunctions);
  }
}

Create health controller

The controller has one endpoint, /health, that injects HeathService to check the readiness of the components.

import { Controller, Get } from '@nestjs/common';
import { HealthCheck, HealthCheckResult } from '@nestjs/terminus';
import { HealthService } from '../services';

@Controller('health')
export class HealthController {
  constructor(private health: HealthService) {}

  @Get()
  @HealthCheck()
  check(): Promise<HealthCheckResult> {
    return this.health.check();
  }
}

Register controller and services

Now, we register new controller and services in HealthModule

// heath.module.ts

import { Module } from '@nestjs/common';
... other import statements ...
import { HealthService } from './services';
import { HealthController } from './controllers';
import { BullQueueHealthIndicator } from './indicators';

@Module({
  imports: [TerminusModule],
  controllers: [HealthController],
  providers: [HealthService, BullQueueHealthIndicator],
  exports: [HealthService],
})
export class HealthModule extends ConfigurableModuleClass {}

Finally, add index.ts to export controllers, interfaces, services, types and the health module

./index.ts

export * from './interfaces';
export * from './services';
export * from './health.module';
export * from './types';

I finished the implementation of the library and will publish it to npmjs.com. Npm publish is not part of the scope of this article.

Use case of health checking

I would continue to use nestjs-health-terminus project to demonstrate how to health check the following resources:

  • backend app
  • Postgres database that uses TypeORM
  • redis
  • bull queues
  • just for fun, ping docs.nestjs.com and angular.io/docs are alive

Clone or fork the Nestjs project

You can find the sample code of nestjs bull queue in this repo: https://github.com/railsstudent/nestjs-health-terminus

First, copy .env.example to .env to load environment variables of redis and Postgres database

REDIS_PORT=6379
REDIS_HOST=localhost
PORT=3000
NODE_DEV=development
BACKEND_DOMAIN=http://localhost:3000
DATABASE_CONNECT=postgres
DATABASE_HOST=localhost
DATABASE_USER=postgres
DATABASE_PASSWORD=postgres
DATABASE_NAME=testDB
DATABASE_PORT=5432

Second, run docker-compose to create and run redis and Postgres

docker-compose up -d

Install dependencies

Install nestjs-health library

npm install --save-exact nestjs-health

Import health module to AppModule

First, create health.config.ts in src/configs folder

import { ConfigService } from '@nestjs/config';
import { HttpHealthIndicator } from '@nestjs/terminus';
import { HealthModule } from 'nestjs-health';

export const healthConfig = HealthModule.forRootAsync({
  inject: [ConfigService, HttpHealthIndicator],
  useFactory: (configService: ConfigService, http: HttpHealthIndicator) => {
    return {
      app: 'nestjs-health-terminus',
      backendUrl: configService.get<string>('BACKEND_DOMAIN', ''),
      shouldCheckDatabase: true,
      queueNames: ['fibonacci', 'prime'],
      redisOptions: {
        host: configService.get<string>('REDIS_HOST', 'localhost'),
        port: configService.get<number>('REDIS_PORT', 0),
      },
      indicatorFunctions: [
        () => http.pingCheck('nestjs-docs', 'https://docs.nestjs.com'),
        () => http.pingCheck('angular-docs', 'https://angular.io/docs'),
      ],
    };
  },
});

heathConfig variable configures dependent resources that require health checking.

Next, import healthConfig in AppModule

@Module({
  imports: [
    ... other modules...
    healthConfig,
  ]
})
export class AppModule {}

Invoke health check endpoint

Run docker-compose to start redis. Since both fibonacci and prime queues can connect to redis, the health check status should be “up”

http://localhost:3000/health

Kill redis and call the same endpoint to display a different response.

docker stop <container id of redis>

redis and bull display health check error messages.

Kill Postgres and call the same endpoint to display a new response.

docker stop <container id of Postgres>

Final Thoughts

In this post, I show how to author a NestJS library to encapsulate health check services and controller in a dynamic module. Then, I publish the library to npmjs and install it into NestJS application to check the health of backend application, database, redis, bull queues and external servers. Finally, the library provides a defined /health endpoint that displays the status of components.

This is the end of the blog post and I hope you like the content and continue to follow my learning experience in NestJS and other technologies.

Resources:

  1. NestJS Heath Library: https://github.com/railsstudent/nestjs-health
  2. Repo: https://github.com/railsstudent/nestjs-health-terminus
  3. npm package: https://www.npmjs.com/package/nestjs-health
  4. Terminus: https://docs.nestjs.com/recipes/terminus
  5. Dynamic Module: https://docs.nestjs.com/fundamentals/dynamic-modules#dynamic-modules

Check health of nestjs bull queue with terminus

Reading Time: 5 minutes

 72 total views,  1 views today

Introduction

It is common for enterprise applications to connect to external resources to process requests and get back responses. Therefore, application should know that the components it depends on are up and running all the times. Otherwise, the application behaves erroneously when serving client requests. To stay aware of the availability of connected resources, we can perform health check on them and wait for the results.

In this blog post, I describe the use case of health check bull queue in nestjs with terminus. First, it requires a custom health indicator to verify that the queue is connected to redis and redis status is ready. Next, I add a /health endpoint that calls the health indicator to return the status.

let's go

Clone or fork the Nestjs project

You can find the sample code of nestjs bull queue in this repo: https://github.com/railsstudent/nestjs-health-terminus

nest generate nestjs-health-terminus

Install dependencies

Install nestjs terminus library to the project

npm install --save @nestjs/terminus

Create health module for health check

nest g mo health

Import HealthModule in AppModule

@Module({
  imports: [
    ... other modules...
    HealthModule,
  ]
})
export class AppModule {}

Add a custom health indicator to health check bull queue

@nestjs/terminus does not provide built-in health indicator for bull queue; therefore, I create a custom one in the health module.

First, Use nestjs CLI to generate a health indicator

nest g s health/health/bullQueue --flat

My convention is to rename the filename suffix from .service.ts to .health.ts.

src/health
├── controllers
│   ├── health.controller.ts
│   └── index.ts
├── health
│   ├── bull-queue.health.ts
│   └── index.ts
├── health.module.ts
└── index.ts

Then, open bull-queue.health.ts and extends BullQueueHealthIndicator from HealthIndicator.

import { getQueueToken } from '@nestjs/bull';
import { Injectable } from '@nestjs/common';
import { ModuleRef } from '@nestjs/core';
import { HealthIndicator, HealthIndicatorResult, HealthCheckError } from '@nestjs/terminus';
import { Queue } from 'bull';

@Injectable()
export class BullQueueHealthIndicator extends HealthIndicator {
  constructor(private moduleRef: ModuleRef) {
    super();
  }
}

Next, import TerminusModule into HealthModule and verify that the providers array has the entry of BullQueueHealthIndicator.

import { Module } from '@nestjs/common';
import { TerminusModule } from '@nestjs/terminus';
import { HealthController } from './controllers';
import { BullQueueHealthIndicator } from './health';

@Module({
  imports: [TerminusModule],
  controllers: [HealthController],
  providers: [BullQueueHealthIndicator],
})
export class HealthModule {}

Add health check logic in custom health indicator

In BullQueueHealthIndicator, I implement isHealthy() method to check the health of fibonacci and prime queues. When the queue connects to redis and redis status is ready, it is considered up. Otherwise, the queue is down and the indicator returns error messages.

async isHealthy(queues: string[]): Promise<HealthIndicatorResult> {
    const promiseResults = await this.checkQueuesReady(queues);
    const errorResults = this.filterErrors(promiseResults);

    if (errorResults.length) {
      throw new HealthCheckError('Bull queue failed', this.getStatus('bull', 
         false, { errors: errorResults }));
    }

    return this.getStatus('bull', true);
 }

checkQueuesReady accepts a list of queue names and examines their readiness.

private async checkQueuesReady(queues: string[]) {
    const promises = queues.map(async (name) => {
      const queueToken = this.moduleRef.get(getQueueToken(name), 
          { strict: false });
      if ((queueToken as Queue).isReady) {
        const queue = await (queueToken as Queue).isReady();
        const isEveryClientReady = queue.clients.every((client) => client.status === 'ready');
        if (!isEveryClientReady) {
          throw new Error(`${name} - some redis clients are not ready`);
        }
        return true;
      }
      throw new Error(`${name} is not a bull queue`);
    });

    return Promise.allSettled(promises);
}

const queueToken = this.moduleRef.get(getQueueToken(name), { strict: false });

The code uses the queue token to obtain the queue instance. When the queue is ready, I get queue.clients that is an array of redis clients. When every redis client has ready status, the queue is ready to accept jobs. On the other hand, when redis store is not set up, the method throws an error message.

private filterErrors(promiseResults: PromiseSettledResult<boolean>[]): string[] {
    const errorResults: string[] = [];
    for (const promiseResult of promiseResults) {
      if (promiseResult.status === 'rejected') {
        if (promiseResult.reason instanceof Error) {
          errorResults.push(promiseResult.reason.message);
        } else {
          errorResults.push(promiseResult.reason);
        }
      }
    }
    return errorResults;
 }

The result of checkQueuesReady is a collection of resolved and rejected promises. filterErrors iterates the promise results, finds those that are rejected and stores the error messages.

When health check returns error messages, isHealthy throws HealthCheckError.

throw new HealthCheckError('Bull queue failed', this.getStatus('bull', false, 
    { errors: errorResults }));

When health check does not find error, isHealthy returns

this.getStatus('bull', true);

Define health controller

nest g co health/controllers/health --flat

import { Controller, Get } from '@nestjs/common';
import { HealthCheck, HealthCheckService } from '@nestjs/terminus';
import { BullQueueHealthIndicator } from '../health/bull-queue.health';

@Controller('health')
export class HealthController {
  constructor(private health: HealthCheckService, private bull: BullQueueHealthIndicator) {}

  @Get()
  @HealthCheck()
  check() {
    return this.health.check([() => this.bull.isHealthy(['fibonacci', 'prime'])]);
  }
}

First, the GET endpoint is annotated with a HealthCheck decorator. Second, the HealthCheckService expects some health indicators to execute. Therefore, I provide BullQueueHealthIndicator to the check method to process.

this.health.check([() => this.bull.isHealthy(['fibonacci', 'prime'])])

checks the health of fibonacci and prime queues and maps the health status to the object with key “bull”.

Invoke health check endpoint

Run docker-compose to start redis. Since both fibonacci and prime queues can connect to redis, the health check status should be “up”

http://localhost:3000/health

Kill docker-compose and call the same endpoint to see a different response.

Both queues are down because redis has not launched yet.

Suppose I also check ‘dummy’ that is not a bull queue, isHealthy outputs bad injection token error message.

return this.health.check([() => this.bull.isHealthy(['fibonacci', 'prime', 'dummy'])]);

Final Thoughts

In this post, I show how to use terminus package to write a custom health indicator to check the health of bull queue. When terminus package does not provide predefined indicators and there is no open source library, we have the ability to write our own indicators to perform the appropriate testing.

This is the end of the blog post and I hope you like the content and continue to follow my learning experience in NestJS and other technologies.

Resources:

  1. Repo: https://github.com/railsstudent/nestjs-health-terminus
  2. Queue: https://docs.nestjs.com/techniques/queues
  3. Bull: https://github.com/OptimalBits/bull
  4. Terminus: https://docs.nestjs.com/recipes/terminus