Build a full-text search with NestJS, MongoDB, Elasticsearch, and Docker (Final Part)
Probably you are not satisfied and convince in the previous section. Cause, we are heading to an application empower to enable a full-text search API.
Continuously to the previous section, in this section, we are going to integrate Elasticsearch service into our application.
Before implementing the Elasticsearch in our application, let’s install all the dependencies needed.
- Install Elasticsearch
npm install --save elasticsearch - Instal Type/Elasticsearch
npm i --save @elastic/elasticsearch - Install Elastic/Elasticsearch
npm i --save @elastic/elasticsearch - Install NestJs/Elasticsearch
npm i --save @nestjs/elasticsearch
Now we are going to implement Elastic service in our application. Elastic is a third-party service that certainly irrelevant in the application’s components.
So, this needs an appropriate design to achieve integration into the application. To gain those ideas, we can create a folder called services which is situated in thesrc folder. Following figure 2.1 illustrate the implementation.
Figure 2.1: Implement search service as a module in NestJS
We can have a look at figure 2.1 above that the search service is implemented as a module in NestJS. Search service will inherit the attributes of Elasticsearch to take its method. Let’s dive dig in the details of the implementation of each file.
search.module.ts
Similar to the other modules in the application. We can declare the search service as a dependency injection in the module.
import { Module } from '@nestjs/common';
import { SearchService } from './search.service';
import { SearchServiceInterface } from './interface/search.service.interface';
@Module({
imports: [],
providers: [
{
provide: 'SearchServiceInterface',
useClass: SearchService
}
],
controllers: [],
exports: [SearchModule]
})
export class SearchModule {}search.service.interface.ts
In the notion of abstraction, we could use an interface to let search service using the reference to the interface with method calls. This could be similar to the other implementation in the application.
Meanwhile, the define methods as insertIndex , updateIndex ..etc in this interface are represent for creating and updating an index into Elasticsearch.
export interface SearchServiceInterface<T> {
insertIndex(bulkData: T): Promise<T>;
updateIndex(updateData: T): Promise<T>;
searchIndex(searchData: T): Promise<T>;
deleteIndex(indexData: T): Promise<T>;
deleteDocument(indexData: T): Promise<T>;
}search.service.ts
And undoubtedly our search service will extend the base Elasticsearch, and implement all methods that declared in the search interface. Some of the methods like a bulk , update have invoked from Elasticsearch.
There is another thing that needs to bear in mind is a configure info with an Elasticsearch URL. We have to define in supper as shown as the snippet below.
import { HttpException, HttpStatus, Injectable } from '@nestjs/common';
import { ElasticsearchService } from '@nestjs/elasticsearch';
import { SearchServiceInterface } from './interface/search.service.interface';
import { ConfigSearch } from './config/config.search';
@Injectable()
export class SearchService extends ElasticsearchService implements SearchServiceInterface<any> {
constructor() {
super(ConfigSearch.searchConfig(process.env.ELASTIC_SEARCH_URL));
}
public async insertIndex(bulkData: any): Promise<any> {
return await this.bulk(bulkData)
.then(res => res)
.catch(err => {
console.log(err);
throw new HttpException(err, HttpStatus.INTERNAL_SERVER_ERROR);
});
}
public async updateIndex(updateData: any): Promise<any> {
return await this.update(updateData)
.then(res => res)
.catch(err => {
throw new HttpException(err, HttpStatus.INTERNAL_SERVER_ERROR);
});
}
public async searchIndex(searchData: any): Promise<any> {
return await this.search(searchData)
.then(res => {
return res.body.hits.hits;
})
.catch(err => {
throw new HttpException(err, HttpStatus.INTERNAL_SERVER_ERROR);
});
}
public async deleteIndex(indexData: any): Promise<any> {
return await this.indices.delete(indexData).then(res => res)
.catch(err => {
throw new HttpException(err, HttpStatus.INTERNAL_SERVER_ERROR);
});
}
public async deleteDocument(indexData: any): Promise<any> {
return await this.delete(indexData).then(res => res)
.catch(err => {
throw new HttpException(err, HttpStatus.INTERNAL_SERVER_ERROR);
});
}
}config.search.ts
export class ConfigSearch {
public static searchConfig(url: string): any {
return {
node: url,
maxRetries: 5,
requestTimeout: 60000,
sniffOnStart: true,
};
}
}product.elastic.ts
We need to set an index and type for every search criteria. In our case, let’s create a product index and type as show as declare below.
export const productIndex = {
_index: 'product',
_type: 'products'
};And finally, we can create a class called product.elastic.index.ts this should contain some of the methods as insert product , update product ..etc helps to inserting and updating the product document into the product Elasticsearch index.
product.elastic.index.ts
import { Inject, Injectable } from '@nestjs/common';
import { SearchServiceInterface } from '../interface/search.service.interface';
import { productIndex } from '../constant/product.elastic';
import { Product } from '../../../components/product/entity/product.entity';
@Injectable()
export class ProductElasticIndex {
constructor(
@Inject('SearchServiceInterface')
private readonly searchService: SearchServiceInterface<any>,
) {
}
public async insertProductDocument(product: Product): Promise<any> {
const data = this.productDocument(product);
return await this.searchService.insertIndex(data);
}
public async updateProductDocument(product: Product): Promise<any> {
const data = this.productDocument(product);
await this.deleteProductDocument(product.id);
return await this.searchService.insertIndex(data);
}
private async deleteProductDocument(prodId: number): Promise<any> {
const data = {
index: productIndex._index,
type: productIndex._type,
id: prodId.toString(),
};
return await this.searchService.deleteDocument(data);
}
private bulkIndex(productId: number): any {
return {
_index: productIndex._index,
_type: productIndex._type,
_id: productId,
};
}
private productDocument(product: Product): any {
const bulk = [];
bulk.push({
index: this.bulkIndex(product.id),
});
bulk.push(product);
return {
body: bulk,
index: productIndex._index,
type: productIndex._type,
};
}
}Those methods in the product.index are paramount important and indispensable when we need to create and update index in the Elasticsearch after inserting or updating events triggered with product collection. Indeed, after creating or updating the product via those two APIs that we have introduced in the previous section, we also need to update our product index in the Elasticsearch.
This implement actually can be invoked into the product service and do some business logic after a product inserting or updating. But there is some disadvantage that we have mentioned in the previous section.
And, observer pattern advent to resolve this problem. We are using TypeORM, this is immeasurably beneficial to implement the observer pattern by the subscriber to product entity and listen to every event happens in product entity like insert, update the product.
We will take a look at figure 2.2 first, revealing the quintessential of the observer pattern and illustrate how’s it works.
Figure 2.2: Illustrate the operation of an observer pattern
Let’s going to implement the observer pattern to listen to the events that happen in the product entity and update the Elasticsearch product index data.
Figure 2.3: Create observers in the application, this exposes as a module.
Let’s have a look at figure 2.3 above, all subscriber class declare into subscribers the folder will listen automatically listen to each defined entity. Make sure that we have a configure in database config that empowered to enabled the subscribers listen automatically. Follow to the snippet below.
ormconfig.ts
subscribers: [
'dist/observers/subscribers/*.subscriber.js'
]Let’s get details into each file.
subscriber.module.ts
import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { Product } from '../components/product/entity/product.entity';
import { SearchService } from '../services/search/search.service';
import { ProductElasticIndex } from '../services/search/search-index/product.elastic.index';
import { SearchServiceInterface } from '../services/search/interface/search.service.interface';
import { PostSubscriber } from './subscribers/product.subscriber';
@Module({
imports: [
TypeOrmModule.forFeature([Product])
],
providers: [
{
provide: 'SearchServiceInterface',
useClass: SearchService
},
ProductElasticIndex,
PostSubscriber
],
controllers: [],
exports: []
})
export class SubscriberModule {}product.subscriber.ts
import { Connection, EntitySubscriberInterface, InsertEvent, UpdateEvent } from 'typeorm';
import { Product } from '../../components/product/entity/product.entity';
import { ProductElasticIndex } from '../../services/search/search-index/product.elastic.index';
import { InjectConnection } from '@nestjs/typeorm';
import { Injectable } from '@nestjs/common';
@Injectable()
export class PostSubscriber implements EntitySubscriberInterface<Product> {
constructor(
@InjectConnection() readonly connection: Connection,
private readonly productEsIndex: ProductElasticIndex) {
connection.subscribers.push(this);
}
public listenTo(): any {
return Product;
}
public async afterInsert(event: InsertEvent<Product>): Promise<any> {
return await this.productEsIndex.insertProductDocument(event.entity);
}
public async afterUpdate(event: UpdateEvent<Product>): Promise<any> {
return await this.productEsIndex.updateProductDocument(event.entity);
}
}Both of SearchModule and ObserverModule have to be imported in the app module.
app.module.ts
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { TypeOrmModule } from '@nestjs/typeorm';
import { ormConfig } from './database/config/ormconfig';
import { ProductModule } from "./components/product/product.module";
import { SearchModule } from "./services/search/search.module";
import { ObserverModule } from "./observers/observer.module";
@Module({
imports: [
ConfigModule.forRoot({
isGlobal: true,
}),
TypeOrmModule.forRoot(ormConfig()),
ProductModule,
SearchModule,
ObserverModule
],
controllers: [],
providers: [],
})
export class AppModule {}Let’s head to implement a full-text search API that allows searching the product by keywords. Follow to the snippets below:
product.module.ts
Declare search service into the provider section of the product module.
{
provide: 'SearchServiceInterface',
useClass: SearchService
}product.service.interface.ts
Adding a new method called search
search(q: any): Promise<any>;Cause we need to prepare object data before making the search. So, let’s create a model called product.search.object.ts and define the snippets below.
product.search.object.ts
import { productIndex } from "../../../services/search/constant/product.elastic";
export class ElasticSearchBody {
size: number;
from: number;
query: any;
constructor(
size: number,
from: number,
query: any
) {
this.size = size;
this.from = from;
this.query = query;
}
}
export class ProductSearchObject {
public static searchObject(q: any) {
const body = this.elasticSearchBody(q);
return { index: productIndex._index, body, q };
}
public static elasticSearchBody(q: any): ElasticSearchBody {
const query = {
match: {
url: q
}
};
return new ElasticSearchBody(
10,
0,
query
);
}
}The product service will inject our search service and implement the search functionality.
product.service.ts
@Inject("SearchServiceInterface")
private readonly searchService: SearchServiceInterface<any>public async search(q: any): Promise<any> {
const data = ProductSearchObject.searchObject(q);
return await this.searchService.searchIndex(data);
}
Finally, we can be exposing our search API functionality to the controller.
product.controller.ts
@Get('/search')
public async search(@Query() query: any): Promise<any> {
return await this.productService.search(query.q);
}The next step, we are going to configure the Elasticsearch service in docker container rely on the docker-compose have configured.
docker-compose.yml
# docker compose version
version: '3.7'
# all the containers have to declare inside services
services:
# App service
demoapp:
# application rely on database running
depends_on:
- db
# this build context will take the commands from Dockerfile
build:
context: .
dockerfile: Dockerfile
# image name
image: nest-demo-docker
# container name
container_name: demoapp
# always restart the container if it stops.
restart: always
# docker run -t is allow
tty: true
# application port, this is take value from env file
ports:
- "${SERVER_PORT}:${SERVER_PORT}"
# working directory
working_dir: /var/www/nest-demo
# application environment
environment:
SERVICE_NAME: demoapp
SERVICE_TAGS: dev
SERVICE_DB_HOST: ${DATABASE_HOST}:${DATABASE_PORT}
SERVICE_DB_USER: ${DATABASE_USERNAME}
SERVICE_DB_PASSWORD: ${DATABASE_PASSWORD}
SERVICE_ES_HOST: ${ELASTIC_SEARCH_HOST}:${ELASTIC_SEARCH_PORT}
ELASTICSEARCH_URL: ${ELASTIC_SEARCH_URL}
# save (persist) data and also to share data between containers
volumes:
- ./:/var/www/nest-demo
- /var/www/nest-demo/node_modules
# application network, each container for a service joins this network
networks:
- nest-demo-network
# Database service
db:
# pull image from docker hub
image: mongo
# container name
container_name: nestmongo
# always restart the container if it stops.
restart: always
# database credentials, this is take value from env file
environment:
MONGO_INITDB_ROOT_DATABASE: ${DATABASE_NAME}
MONGO_INITDB_ROOT_USERNAME: ${DATABASE_USERNAME}
MONGO_INITDB_ROOT_PASSWORD: ${DATABASE_PASSWORD}
# save (persist) data and also to share data between containers
volumes:
- db_data:/data/db
# database port
ports:
- "${DATABASE_PORT}:${DATABASE_PORT}"
# application network, each container for a service joins this network
networks:
- nest-demo-network
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.0
container_name: elasticsearch
environment:
- node.name=elasticsearch
- http.port=9200
- http.host=0.0.0.0
- transport.host=127.0.0.1
- cluster.name=es-docker-cluster
- discovery.seed_hosts=elasticsearch
- cluster.initial_master_nodes=elasticsearch
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es_data:/var/lib/elasticsearch
ports:
- "${ELASTIC_SEARCH_PORT}:${ELASTIC_SEARCH_PORT}"
networks:
- nest-demo-network
#Docker Networks
networks:
# All container connect in a network
nest-demo-network:
driver: bridge
# save (persist) data
volumes:
db_data: {}
es_data: {}.env the file also has some more variables needed.
ELASTIC_SEARCH_URL=http://elasticsearch:9200/
ELASTIC_SEARCH_HOST=elasticsearch
ELASTIC_SEARCH_PORT=9200And finally, now we can be running up our application via the command docker-compose up . After that, we can perform some testing in the postman as those figures shown below.
Figure 2.5: Perform create a product, this will also insert to Elasticsearch index after API invoked.
Figure 2.6: Search the product by keyword test just inserted
Figure 2.7: Perform update a created product, this will also update to Elasticsearch index after API invoked.
Figure 2.8: Search the product by keyword another just updated.
In conclusion, we have built an application that exposes a full-text search API using Elasticsearch and the observer pattern to trigger inserting, updating a product index into Elasticsearch data.
Thanks for reading!
Github source: https://github.com/phatvo21/nestjs-elastic-search-docker
List of content:
