MongoDB real-time sync with Elasticsearch using monstache in Docker

I used Kibana to provide a solution to visualize the data of an OMS, but I was looking for a way to synchronize real time collections from MongoDB to Elasticsearch to offer a real time data visualization; after having tried with several libraries, I found.

Monstache

“Monstache is a sync daemon written in Go that continously indexes your MongoDB collections into Elasticsearch. It is easy to configure using docker.”

First of all, create the following directories and files:

To config the container, copy into docker-compose.yml file:

version: '3' 
    services:
        monstache_container_name:        
            image: rwynn/monstache:latest        
            container_name: container_name       
            working_dir: /monstache        
            command: -f ./config.toml        
            volumes:          
                - ./monstache:/monstache/        
            ports:          
                - "8080:8080"        
            restart: always        
networks:  
    app-network:    
        driver: bridge

To config monstache pipeline copy into config.toml file:

mongo-url = "mongodb://[username:password@]host1[:port1]" 
elasticsearch-urls = ["http://cluster_id.us-west-5.aws.found.io:9200"]
elasticsearch-user = "elastic"
elasticsearch-password = "secret"
direct-read-namespaces = ["databasename.Collection"]
change-stream-namespaces = ["databasename.Collection"] 
[logs]
error = "./logs/error.log" 
[[mapping]]
namespace = "databasename.Collection" 
index = "IndexName"
 
[[script]]
namespace = "databasename.Collection"
path = "./scripts/collection_name.js"
routing = true

Set your

• mongodb-url
• elasticsearch-url
• elasticsearch credential, user and password.

direct-read-namespaces: This option allows you to directly copy collections from MongoDB to Elasticsearch. Monstache allows filtering the data that is actually indexed to Elasticsearch, so you not necessarily need to copy the entire collection.

change-stream-namspace: Notify in real-time to elasticsearch of all writes documents including deletes and updates in the specified collections.

[logs]: Log the errors, useful for issues.

[mapping]: Override the default index name.

[[script]]: Is a middleware that is able to transform, drop documents, or define indexing metadata. This middleware can be written in either Javascript or in Golang as a plugin.

Example of script to transform documents.

module.exports = function (doc) {
    delete doc._id;
    //TODO
    return doc;
}

Copy into collection_name.js file.

Then, to run the container use the command

~$ docker-compose up -d