MongoDB real-time sync with Elasticsearch using monstache in Docker

Andrés Echeverry
Jul 24 · 2 min read

I used Kibana to provide a solution to visualize the data of an OMS, but I was looking for a way to synchronize real time collections from MongoDB to Elasticsearch, after having tried with several libraries, I found.

Monstache is a sync daemon written in Go that continously indexes your MongoDB collections into Elasticsearch. It is easy to configure using docker.

First of all, create the following directories and files:

Image for post
Image for post

To config the container, copy into docker-compose.yml file:

To config monstache pipeline copy into config.toml file:

Set your

  • mongodb-url
  • elasticsearch-url
  • elasticsearch credential, user and password.

direct-read-namespaces: This option allows you to directly copy collections from MongoDB to Elasticsearch. Monstache allows filtering the data that is actually indexed to Elasticsearch, so you not necessarily need to copy the entire collection.

change-stream-namspace: Notify in real-time to elasticsearch of all writes documents including deletes and updates in the specified collections.

[logs]: Log the errors, useful for issues.

[mapping]: Override the default index name.

[[script]]: Is a middleware that is able to transform, drop documents, or define indexing metadata. This middleware can be written in either Javascript or in Golang as a plugin.

Example of script to transform documents.

Copy into collection_name.js file.

Then, to run the container use the command

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store