Predicting Machine Failure with Time-Series Pattern Matching
Time-series data is a roadmap of the past, a tool for the present, and a forecast of the future.
An Introduction to Temporal Similarity Search (TSS)
As the volume of time-series data continues to surge, organizations face the challenge of turning this data into actionable insights. Temporal Similarity Search, a built-in algorithm of the KDB.AI vector database, enables you to easily understand patterns, trends, and anomalies in time-series data without requiring the use of more complex algorithms and machine learning tactics.
In this tutorial, I’ll demonstrate how to ingest time-series data — whether from manufacturing, IoT, financial, or other sources — into KDB.AI and build a near real-time pattern-matching pipeline. This pipeline will help you uncover patterns in your data, predict maintenance needs, enhance quality control, and much more.
In our example code, we will focus on a specific version of TSS called ‘Transformed Temporal Similarity Search’. Transformed TSS is a patent-pending compression model that reduces the dimensionality of time series windows by over 99%, while preserving the original data’s shape. These compressed windows are stored as vector embeddings in KDB.AI, facilitating highly efficient vector searches across extensive time series collections. This enables effective analysis and rapid retrieval of patterns and anomalies within large-scale time series data.
Applications of Time-Series Pattern Matching
Time-series data, when effectively analyzed, can unlock significant opportunities for improving operational efficiency and reliability. Three example applications of TSS are predictive maintenance, quality control, and financial market trend analysis. Let’s have a quick discussion on each.
Predictive Maintenance
Predictive maintenance involves using TSS to predict when equipment failures might occur. By analyzing patterns in time-series data from sensors, it is possible to find early signs of equipment wear and tear or malfunctions. This means that maintenance can be scheduled proactively, reducing downtime and extending the lifetime of machinery. For instance, if a particular pattern has historically preceded a failure, detecting the pattern early can trigger a maintenance alert before the failure occurs.
Quality Control
Quality control ensures that products meet specified standards and are free from defects. TSS provides the capability to monitor the production process in real-time and identify anomalies that might indicate quality issues. Early detection enables prompt intervention, ensuring fast fixes to the production equipment and only high quality products reaching the market.
Market Trend Analysis
Financial markets generate vast amounts of time series data, including stock prices, trading volumes, and economic indicators. TSS can enable analysts to identify recurring patterns and trends within this data. For instance, TSS can detect bullish or bearish trends by comparing current market conditions with historical periods that exhibited similar characteristics.
Let’s Build It!
What we are doing: In this tutorial, we will take IoT sensor data, and use pattern matching to identify when the sensor is approaching a failed state.
See the full notebook on our GitHub.
Or, open the code directly in Google Colab.
Prerequisites & Environment Setup
First, we need to acquire a KDB.AI endpoint and API key: https://kdb.ai/get-started.
Next, install dependencies:
!pip install kdbai_client matplotlibImport Packages:
# read data
from zipfile import ZipFile
import pandas as pd
# plotting
import matplotlib.pyplot as plt
# vector DB
import os
import kdbai_client as kdbai
from getpass import getpass
import timeLoad & Prepare Time-Series Data
The dataset that will be used for this example is the Water Pump Sensor Dataset available on Kaggle. The dataset consist of a `sensor.csv` file which has raw values from 52 sensors from a town water pump.
# Download and unzip our time-series sensor dataset
def extract_zip(file_name):
with ZipFile(file_name, "r") as zipf:
zipf.extractall("data")
extract_zip("data/archive.zip")
# Read in our dataset as a Pandas DataFrame
raw_sensors_df = pd.read_csv("data/sensor.csv")
show_df(raw_sensors_df)
Preprocess the Dataset
# Drop duplicates
sensors_df = raw_sensors_df.drop_duplicates()
# Remove columns that are unnecessary/bad data
sensors_df = sensors_df.drop(["Unnamed: 0", "sensor_15", "sensor_50"], axis=1)
# convert timestamp to datetime format
sensors_df["timestamp"] = pd.to_datetime(sensors_df["timestamp"])
# Removes rows with any NaN values
sensors_df = sensors_df.dropna()
# Reset the index
sensors_df = sensors_df.reset_index(drop=True)Explore the Data for Sensor_00
There is a column in the dataframe called “machine_status” that holds information on when the sensor is reading a “BROKEN” status. Let’s plot this alongside the sensor reading to visualize the pattern.
# Extract the readings from the BROKEN state of the pump
broken_sensors_df = sensors_df[sensors_df["machine_status"] == "BROKEN"]
# Plot time series for each sensor with BROKEN state marked with X in red color
plt.figure(figsize=(18, 3))
plt.plot(
broken_sensors_df["timestamp"],
broken_sensors_df["sensor_00"],
linestyle="none",
marker="X",
color="red",
markersize=12,
)
plt.plot(sensors_df["timestamp"], sensors_df["sensor_00"], color="blue")
plt.show()
We see that the blue line, which represents the sensor signal, hovers around 2.5, but occasionally drops to between 0 and 0.5. Typically when this happens, the machine status is broken as identified by the red ‘x’
For this solution, we will specifically evaluate sensor_00, so we only need that sensor in our DataFrame (for reference it contains 195,815 rows):
sensor0_df = sensors_df[["timestamp", "sensor_00"]]
sensor0_df = sensor0_df.reset_index(drop=True).reset_index()
# This is our sensor data to be ingested into KDB.AI
sensor0_df.head()
Group the Sensor Values into Time Windows
# Set the window size (number of rows in each window)
window_size = 100
step_size = 1
# define windows
windows = [
sensor0_df.iloc[i : i + window_size]
for i in range(0, len(sensor0_df) - window_size + 1, step_size)
]
# Iterate through the windows & extract column values
start_times = [w["timestamp"].iloc[0] for w in windows]
end_times = [w["timestamp"].iloc[-1] for w in windows]
sensor0_values = [w["sensor_00"].tolist() for w in windows]
# Create a new DataFrame from the collected data
embedding_df = pd.DataFrame(
{"timestamp": start_times, "sensor_00": sensor0_values}
)
embedding_df = embedding_df.reset_index(drop=True).reset_index()
embedding_df.head()
Now each data point has a 100 dimension time window associated with it (as seen in column ‘sensor_00’). This time window starts with the sensor value at the timestamp of that column, as well as 99 values ahead of it. This DataFrame can now be inserted into KDB.AI.
Store Time-Series Windows in KDB.AI
You can get a KDB.AI API key and endpoint for free here: https://trykdb.kx.com/kdbai/signup/
KDBAI_ENDPOINT = (
os.environ["KDBAI_ENDPOINT"]
if "KDBAI_ENDPOINT" in os.environ
else input("KDB.AI endpoint: ")
)
KDBAI_API_KEY = (
os.environ["KDBAI_API_KEY"]
if "KDBAI_API_KEY" in os.environ
else getpass("KDB.AI API key: ")
)
session = kdbai.Session(api_key=KDBAI_API_KEY, endpoint=KDBAI_ENDPOINT)You are now connected to the vector database instance — the next step is to define the schema for the table you will create within KDB.AI:
sensor_schema = dict(
columns=[
dict(
name='index',
pytype='int64'
),
dict(
name='timestamp',
pytype='datetime64[ns]'
),
dict(
name='sensor_00',
pytype='float64',
vectorIndex=
dict(
type='flat',
metric='L2',
),
embedding=
dict(
dims=8,
type='tsc', # 'tsc' specifies to KDB.AI to use Transformed TSS
)
)
]
)Here we are just defining the columns that will be in our table.
- index: Used as a unique identifier for each time series window and to help us understand where within the signal that a pattern is located.
- timestamp: The timestamp of the first value in the time series window for that row/index.
- sensor_00: This is the column where the magic (TSS) happens, here we specify the vector index type (flat), and search metric (L2 — Euclidean Distance), as well as the the embedding dims (8, compressing the 100 dimension windows into 8 dimensions for faster and more memory efficient similarity search!), and finally the type (‘tsc’ — specifying we are using Transformed Temporal Similarity Search to search our dataset)
Now we can create the table using the above schema, and insert our data!
# First ensure the table does not already exist
try:
session.table("sensors").drop()
time.sleep(5)
except kdbai.KDBAIException:
pass
table = session.create_table("sensors", sensor_schema)
# Insert our data into KDB.AI
from tqdm import tqdm
n = 1000 # number of rows per batch
for i in tqdm(range(0, embedding_df.shape[0], n)):
table.insert(embedding_df[i:i+n].reset_index(drop=True))🎉Wooo!! We have inserted our time series windows into KDB.AI and are now ready to find similar patterns with Transformed TSS!
Perform Temporal Similarity Search to Identify Equipment Failures
Our goal is to identify the first failure event, and use the time window around that event as our ‘query vector’. We will then search the rest of our dataset with that pattern to try to identify the other instances of a broken machine.
Let’s take a quick look at when our sensor is reading a machine failure:
broken_sensors_df["timestamp"]
We see the first broken index is at 17,125, so to capture the moments before the failure, we will take our query vector to be index 17,100.
# This is our query vector, using the 17100th window as an example
# (this is just before the first instance when the sensor is in a failed state)
q = embedding_df['sensor_00'][17100]Let’s visualize the query vector:
# Visualise the query pattern
plt.figure(figsize=(10, 6))
plt.plot(embedding_df['sensor_00'][17100], marker="o", linestyle="-")
plt.xlabel("Timestamp")
plt.ylabel("Value")
plt.title("Query Vector")
plt.grid(True)
plt.xticks(rotation=45) # Rotate x-axis labels for readability
plt.show()
Execute search to retrieve the top 100 similar matches to see if we can identify the other instances of broken machines (we do metadata filtering to only return windows that are past the first failure):
nn1_result = table.search([q], n=100, filter=[(">","index", 18000)])
nn1_result[0]
As we can see in the first two matches are indexes 64,949 and 64,948. Since every timestamp has a 100 dimension window, it is expected that we will have some matches that are close to one another, but we need to understand that these matches are overlapping and represent the same underlying pattern. To ensure we are only returning unique pattern matches, we will implement some filtering to remove any matches within a range of 200 data points from our next closest match. This means that index 64,949 will be included as a final result, but 64,948 will be excluded because it is within 200 data points of 64,949 and therefore it’s pattern is already accounted for.
def filter_results(df, range_size=200):
final_results = []
removed_indices = set()
for _, row in df.iterrows():
current_index = row['index']
# If this index hasn't been removed
if current_index not in removed_indices:
final_results.append(row)
# Mark indices within range for removal
lower_bound = max(0, current_index - range_size // 2)
upper_bound = current_index + range_size // 2
removed_indices.update(range(lower_bound, upper_bound + 1))
# Create a new dataframe from the final results
final_df = pd.DataFrame(final_results)
return final_df
filtered_df = filter_results(nn1_result[0])
# Display the filtered results
print(filtered_df)The final filtered results identifying all similar patterns to our query vector — in other words, each result here should represent when the sensor is reading a broken machine:
Display our broken sensors again for comparison:
broken_sensors_df["timestamp"]Results: We see that our final results closely capture each of the timestamps of the failed states within a few indexes. There is only one captured pattern that is not within the failed states: 110,667. If you take another look at the original signal you will see within the pattern a large drop in the signal near this index (highlighted below)— this could show a time that needs to be investigated as a potential missed failed state.
Visualize the Retrieved Patterns
Let’s visualize the pattern for each of the final results. We see that each signal has a large drop from around 2.5 to between 0 and 0.5, signifying a broken machine.
for i in filtered_df['index']:
plt.plot(embedding_df['sensor_00'][i], marker="o", linestyle="-")
plt.xlabel("Timestamp")
plt.ylabel("Value")
plt.title("Query & Similar Patterns")
plt.grid(True)
plt.xticks(rotation=45) # Rotate x-axis labels for readability
plt.show()
With Transformed TSS, we were able to identify the other instances of a failing machine in our time series dataset without the need to implement any complex statistical methods or machine learning. I encourage you to take this notebook and sculpt it to your own time-series use-cases!
Further Exploration
- Read the Temporal Similarity Search documentation
- A YouTube Tutorial of this notebook
- Find further samples, demos, and articles on the KDB.AI Learning Hub
Connect with me on LinkedIn
