Building a Milvus Connector For NiFi

Apache NiFi, Python, Milvus, Zilliz, Cloudera, Vector Database

If you are interested in Milvus with Enterprise Data and Enterprise AI, join us on May 1st, 2024 (or later via Youtube) in this awesome AI + RAG + Milvus + NiFi + LLM + GenAI Event.

Gen AI on Enterprise Cloud · Luma

Generative AI on Enterprise Cloud with NiFi and Milvus

For the first step, we will test a simple Python script using pymilvus 2.4 to insert Medium articles into our Milvus database.

Source Code

GitHub - tspannhw/FLaNK-Milvus: Flink - NiFi - Kafka - Milvus - Zilliz -Cloudera

Example Result

{
  "uuid" : "52e64557-fee1-4575-86bd-6b2d339c4f1a",
  "systemtime" : "04/02/2024 21:02:50",
  "ts" : 1712106170,
  "milvusdbname" : "_default",
  "milvusinsertresult" : "(insert count: 1, delete count: 0, upsert count: 0, timestamp: 448818359205888002, success count: 1, err count: 0)",
  "milvuscollectioname" : "MilvusDocs",
  "milvusdocsource" : "Medium",
  "maxseqlength" : "512",
  "embeddinglength" : "768"
}

Attu Milvus Display

Slack Output

Apache NiFi Python Processor

This is in development now.

RESOURCES

Install Milvus Standalone with Docker

Insert Entities

Conduct a Vector Similarity Search

Milvus Backup

GitHub - milvus-io/milvus-insight: Milvus management GUI

GitHub - zilliztech/attu: The GUI for Milvus

Zilliz Cloud Supports Models from OSS, VoyageAI, and OpenAI - Zilliz blog

How to Enhance the Performance of Your RAG Pipeline - Zilliz blog