Member-only story

Top 3 Questions to Ask in Near Real-Time Data Solutions

Questions that guide architectural decisions to balance functional requirements like most frequent queries with non-functional requirements such as latency and scalability.

Shawn Shi
TDS Archive
16 min readJan 16, 2025

--

Image by author. Icons from flaticon.com.

Introduction

Would you agree that our life is surrounded by technology nowadays? Technology has changed our lifestyle in all kinds of ways in the past decade. Behind the scene, data is the core to almost every technology. In our everyday life, for example, e-commerce platforms make personalized product recommendations by tracking our user behavior, and IoT systems like thermostats in our house collect data to smartly adjust temperature settings. This means a massive amount of data needs to be collected and processed constantly. How do we handle such large amounts of data in a real-time or near real-time manner? One key challenge is to design efficient and scalable data solutions.

Let’s explore a few common scenarios before we dive into their challenges:

  • Log Analytics: Monitoring application logs, such as web request HTTP status codes, system metrics like CPU usage, and application error events. Some example data queries can be “What are the top 10 error codes logged in the past 24…

--

--

TDS Archive
TDS Archive

Published in TDS Archive

An archive of data science, data analytics, data engineering, machine learning, and artificial intelligence writing from the former Towards Data Science Medium publication.

Shawn Shi
Shawn Shi

Written by Shawn Shi

Senior Software Engineer at Microsoft. Ex-Machine Learning Engineer. When I am not building applications, I am playing with my kids or outside rock climbing!

Responses (1)