AI Financial Assistant with data streaming, SingleStore pipelines and advanced RAG (Part1)

TLDR

• We are going to stream “live” financial market data with dxFeed API and Redpanda
• Store SEC 10K financial reports on AWS S3
• Consume data using SingleStore pipelines

• Also, we are going to use Docker containers to spin up services
• Use Terraform and easily deploy Redpanda broker/producer on any cloud
• For now, I have the option to choose between AWS and GCP, but could easily expand to other Clouds (Azure, Alibaba, IBM, DigitalOcean, etc.)

• After that, in Part 2 we will use multiple LLMs, Lanchain, Llamaindex and advanced RAG techniques to interact with this live data
• In Part 3 (if we have enough demand), we could build the frontend, deploy it to production, add monitoring, CI/CD pipelines and explore other options to make it production-ready

For the rest of the post, visit:

https://mpdev.hashnode.dev/ai-financial-assistant-with-data-streaming-singlestore-pipelines-and-advanced-rag-part1

Feel free to contact me for consulting and development and book a call at: https://calendly.com/agihive/30min

Originally published at https://mpdev.hashnode.dev.