What’s in the Air Tonight Mr. Milvus?

Open Source, Air Quality, REST, JSON, Python, Milvus, Vector Database

What is in the air tonight? The only way to find out is to use existing air quality data feeds or gather your own with sensors. Today we will use existing public air quality readings from many locations. We can also target specific locations or grab locations on demand.

Boston, Where’s My Bus? LLM + Streaming to the Rescue

Home

OpenAQ is a nonprofit organization providing universal access to air quality data to empower a global community of changemakers to solve air inequality — the unequal access to clean air.

They provide a great source of Air Quality data so you know about where you are or are going.

Introduction

We can do some historical analysis with GZipped CSVs.

Accessing OpenAQ Archive Data

You will need to register for an account.

Create an account

They even have great developer documentation.

Recipes

The other main source besides sensors is AirNow, which is a US government site.

Air Quality Index (AQI)

We did a quick example of that.

AirNow.gov

Constants and Database Connection

import os
from pymilvus import MilvusClient
from dotenv import load_dotenv
load_dotenv(verbose=True)

DIMENSION = 384 
MILVUS_URL = "http://192.168.1.1:19530" 
COLLECTION_NAME = "airquality"
AQ_URL = "https://www.airnowapi.org/aq/observation/zipCode/current/?format=application/json&distance=5000&zipCode="
AQ_KEY = "&API_KEY="
API_KEY = "value"
milvus_client = MilvusClient( uri=MILVUS_URL )

Embedding for Small Text

import json
import requests
from pymilvus import model
from pymilvus.model.dense import SentenceTransformerEmbeddingFunction
model = SentenceTransformerEmbeddingFunction('all-MiniLM-L6-v2',device='cpu' )

Vector Database Setup

from pymilvus import connections
from pymilvus import utility
from pymilvus import FieldSchema, CollectionSchema, DataType, Collection
import pprint

schema = milvus_client.create_schema(
    enable_dynamic_field=False
)

schema.add_field(field_name='id', datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name='DateObserved', datatype=DataType.VARCHAR, max_length=12)
schema.add_field(field_name="HourObserved", datatype=DataType.INT64)
schema.add_field(field_name="Latitude", datatype=DataType.FLOAT)
schema.add_field(field_name="Longitude", datatype=DataType.FLOAT)
schema.add_field(field_name='ParameterName', datatype=DataType.VARCHAR, max_length=16)
schema.add_field(field_name='ZipCode', datatype=DataType.VARCHAR, max_length=16)
schema.add_field(field_name="AQI", datatype=DataType.FLOAT)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=DIMENSION)
schema.add_field(field_name="details",  datatype=DataType.VARCHAR, max_length=8000)
schema.add_field(field_name="location",  datatype=DataType.VARCHAR, max_length=2000)

index_params = milvus_client.prepare_index_params()
index_params.add_index(
    field_name="id",
    index_type="STL_SORT"
)

index_params.add_index(
    field_name="vector",
    index_type="IVF_FLAT",
    metric_type="L2",
    params={"nlist": 100}
)

if milvus_client.has_collection(collection_name=COLLECTION_NAME):
    print("Exists.")
else:
    milvus_client.create_collection(
        collection_name = COLLECTION_NAME,
        schema=schema,
        index_params=index_params
    )
    
    res = milvus_client.get_load_state(
        collection_name = COLLECTION_NAME
    )
    print(res)

Data Engineering Ingest

import numpy
zipcodes = ["08520","94027","02199","11962","94957","90402","94301","07070","90265","90272","10013","10007","94123","77449","11368","60629","79936","75034"]
data = []
for i, val in enumerate(zipcodes):
    response = requests.get(str(AQ_URL + val + AQ_KEY + API_KEY )).content
    airqualities = json.loads(response)
    
    details = ""
    location = ""
    
    for jsonitems in airqualities:
         print(val)
         location = 'Location {0} {1} @ {2},{3}'.format(jsonitems.get('ReportingArea','North'),  jsonitems.get('StateCode','NY'), jsonitems.get('Latitude',0),jsonitems.get('Longitude',0)  )
         details = 'Current Air Quality Reading for {0} is {1} for {2} at Hour {3} on {4}.'.format( jsonitems.get('ParameterName','O3'),jsonitems.get('AQI',0), location, jsonitems.get('HourObserved',1),jsonitems.get('DateObserved','2024-01-01'))
    
         data.append({"DateObserved": jsonitems.get('DateObserved',''), 
                      "HourObserved": jsonitems.get('HourObserved',0), 
                      "Latitude": jsonitems.get('Latitude',0), 
                      "Longitude": jsonitems.get('Longitude',0), 
                      "ParameterName": jsonitems.get('ParameterName',''), 
                      "ZipCode": str(val),
                      "AQI": jsonitems.get('AQI',0.0), 
                      "vector": model(details), "details": details, "location": location})

res = milvus_client.insert(collection_name=COLLECTION_NAME, data=data)

Search Code

search_params = {
    "metric_type": "L2",
    "params": {"nprobe": 10}
}
query_vector = [data[0]["vector"]]
search_results = milvus_client.search(
    COLLECTION_NAME,
    data=query_vector,
    anns_field="vector",
    output_fields=["id", "DateObserved", "HourObserved", "Latitude", "Longitude", "ParameterName", "ZipCode", "AQI", "details", "location"],
    search_params=search_params,
    limit=10
)
for hits in search_results:
    for hit in hits:
        print(f"Hit: {hit}")

Search Results

Hit: {'id': 451730705222766231, 'distance': 0.0, 'entity': {'HourObserved': 15, 'Longitude': -74.32499694824219, 'ParameterName': 'O3', 'AQI': 32.0, 'details': 'Current Air Quality Reading for O3 is 32 for Location Central NJ @ 40.401,-74.325 at Hour 15 on 2024-08-17.', 'id': 451730705222766231, 'Latitude': 40.4010009765625, 'ZipCode': '08520', 'location': 'Location Central NJ @ 40.401,-74.325', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766244, 'distance': 0.05379501357674599, 'entity': {'HourObserved': 15, 'Longitude': -74.18699645996094, 'ParameterName': 'O3', 'AQI': 46.0, 'details': 'Current Air Quality Reading for O3 is 46 for Location Northeast Urban NJ @ 40.692,-74.187 at Hour 15 on 2024-08-17.', 'id': 451730705222766244, 'Latitude': 40.69200134277344, 'ZipCode': '07070', 'location': 'Location Northeast Urban NJ @ 40.692,-74.187', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766249, 'distance': 0.05379501357674599, 'entity': {'HourObserved': 15, 'Longitude': -74.18699645996094, 'ParameterName': 'O3', 'AQI': 46.0, 'details': 'Current Air Quality Reading for O3 is 46 for Location Northeast Urban NJ @ 40.692,-74.187 at Hour 15 on 2024-08-17.', 'id': 451730705222766249, 'Latitude': 40.69200134277344, 'ZipCode': '10013', 'location': 'Location Northeast Urban NJ @ 40.692,-74.187', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766251, 'distance': 0.05379501357674599, 'entity': {'HourObserved': 15, 'Longitude': -74.18699645996094, 'ParameterName': 'O3', 'AQI': 46.0, 'details': 'Current Air Quality Reading for O3 is 46 for Location Northeast Urban NJ @ 40.692,-74.187 at Hour 15 on 2024-08-17.', 'id': 451730705222766251, 'Latitude': 40.69200134277344, 'ZipCode': '10007', 'location': 'Location Northeast Urban NJ @ 40.692,-74.187', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766238, 'distance': 0.1820998340845108, 'entity': {'HourObserved': 15, 'Longitude': -72.55059814453125, 'ParameterName': 'O3', 'AQI': 40.0, 'details': 'Current Air Quality Reading for O3 is 40 for Location Madison CT @ 41.2583,-72.5506 at Hour 15 on 2024-08-17.', 'id': 451730705222766238, 'Latitude': 41.25830078125, 'ZipCode': '11962', 'location': 'Location Madison CT @ 41.2583,-72.5506', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766256, 'distance': 0.24976064264774323, 'entity': {'HourObserved': 15, 'Longitude': -73.83589935302734, 'ParameterName': 'O3', 'AQI': 46.0, 'details': 'Current Air Quality Reading for O3 is 46 for Location New York City Region NY @ 40.8419,-73.8359 at Hour 15 on 2024-08-17.', 'id': 451730705222766256, 'Latitude': 40.84189987182617, 'ZipCode': '11368', 'location': 'Location New York City Region NY @ 40.8419,-73.8359', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766235, 'distance': 0.25786182284355164, 'entity': {'HourObserved': 15, 'Longitude': -71.0510025024414, 'ParameterName': 'O3', 'AQI': 28.0, 'details': 'Current Air Quality Reading for O3 is 28 for Location Boston MA @ 42.351,-71.051 at Hour 15 on 2024-08-17.', 'id': 451730705222766235, 'Latitude': 42.35100173950195, 'ZipCode': '02199', 'location': 'Location Boston MA @ 42.351,-71.051', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766241, 'distance': 0.265556663274765, 'entity': {'HourObserved': 12, 'Longitude': -118.45659637451172, 'ParameterName': 'O3', 'AQI': 35.0, 'details': 'Current Air Quality Reading for O3 is 35 for Location NW Coastal LA CA @ 34.0505,-118.4566 at Hour 12 on 2024-08-17.', 'id': 451730705222766241, 'Latitude': 34.050498962402344, 'ZipCode': '90402', 'location': 'Location NW Coastal LA CA @ 34.0505,-118.4566', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766248, 'distance': 0.265556663274765, 'entity': {'HourObserved': 12, 'Longitude': -118.45659637451172, 'ParameterName': 'O3', 'AQI': 35.0, 'details': 'Current Air Quality Reading for O3 is 35 for Location NW Coastal LA CA @ 34.0505,-118.4566 at Hour 12 on 2024-08-17.', 'id': 451730705222766248, 'Latitude': 34.050498962402344, 'ZipCode': '90272', 'location': 'Location NW Coastal LA CA @ 34.0505,-118.4566', 'DateObserved': '2024-08-17'}}
Hit: {'id': 451730705222766253, 'distance': 0.2820756137371063, 'entity': {'HourObserved': 12, 'Longitude': -122.43000030517578, 'ParameterName': 'O3', 'AQI': 12.0, 'details': 'Current Air Quality Reading for O3 is 12 for Location San Francisco CA @ 37.75,-122.43 at Hour 12 on 2024-08-17.', 'id': 451730705222766253, 'Latitude': 37.75, 'ZipCode': '94123', 'location': 'Location San Francisco CA @ 37.75,-122.43', 'DateObserved': '2024-08-17'}}

AIM-AirQuality/airquality.ipynb at main · tspannhw/AIM-AirQuality

Ok now it’s time for the main event, utilizing OpenAQ data for the world for Q&A on Air Quality utilizing Milvus, LangChain, Python and some libraries. Let’s go.

DATA ENGINEERING — ACQUISITION

We load from the REST end point, in future articles I will show you how to load from sensors.

See: https://github.com/tspannhw/AIM-AirQuality/blob/main/aqmetricsload.py

import json
import requests
from pymilvus import model
from pymilvus.model.dense import SentenceTransformerEmbeddingFunction
model = SentenceTransformerEmbeddingFunction(‘all-MiniLM-L6-v2’,device=’cpu’ )
DIMENSION = 384
MILVUS_URL = “http://localhost:19530"
COLLECTION_NAME = “airquality”
AQ_URL = “https://www.airnowapi.org/aq/observation/zipCode/current/?format=application/json&distance=5000&zipCode="
AQ_KEY = “&API_KEY=”
API_KEY = os.environ.get(“API_KEY”)slack_token = os.environ.get(“SLACK_BOT_TOKEN”)
client = WebClient(token=slack_token)
milvus_client = MilvusClient( uri=MILVUS_URL )

from datetime import datetime, timedelta

OAQ3_COLLECTION_NAME = “openaqmeasurements”
TODAYS_DATE = str( datetime.today().strftime(‘%Y-%m-%d’) )
YESTERDAYS_DATE = (datetime.now() — timedelta(1)).strftime(‘%Y-%m-%d’)

In the first section we have some imports (see the full code for all the imports, there are a lot of Python libraries required). We also have some constants and variables for connecting to Milvus and our external data.

We are also setting today’s date and yesterday’s date for getting a range of data from the API.

We build up a schema for all of our fields and vector. As you can see you can have a lot of fields in your collection.

schema = milvus_client.create_schema(
enable_dynamic_field=False
)

schema.add_field(field_name=’id’, datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name=’locationId’, datatype=DataType.INT32)
schema.add_field(field_name=’location’, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=’parameter’, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=”value”, datatype=DataType.FLOAT)
schema.add_field(field_name=’datelocal’, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=”unit”, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=”latitude”, datatype=DataType.FLOAT)
schema.add_field(field_name=”longitude”, datatype=DataType.FLOAT)
schema.add_field(field_name=”country”, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=”city”, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=”isMobile”, datatype=DataType.VARCHAR, max_length=12)
schema.add_field(field_name=”isAnalysis”, datatype=DataType.VARCHAR, max_length=12)
schema.add_field(field_name=’entity’, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=’sensorType’, datatype=DataType.VARCHAR, max_length=255)
schema.add_field(field_name=”vector”, datatype=DataType.FLOAT_VECTOR, dim=DIMENSION)
schema.add_field(field_name=”details”, datatype=DataType.VARCHAR, max_length=8000)
schema.add_field(field_name=”fulllocation”, datatype=DataType.VARCHAR, max_length=2000)

We add fields for the all the fields returned by API, plus a few fields we build up on our own. The field details is a description of the Air Quality build with all the fields. We then use the embedding function to encode against the details.

If it doesn’t exist all ready we build some indexes and create our collection.

index_params = milvus_client.prepare_index_params()

index_params.add_index(
field_name=”id”,
index_type=”STL_SORT”
)

index_params.add_index(
field_name=”vector”,
index_type=”IVF_FLAT”,
metric_type=”L2",
params={“nlist”: 100}
)

if not milvus_client.has_collection(collection_name = OAQ3_COLLECTION_NAME):
milvus_client.create_collection(
collection_name = OAQ3_COLLECTION_NAME,
schema=schema,
index_params=index_params
)

res = milvus_client.get_load_state(
collection_name = OAQ3_COLLECTION_NAME
)
print(res)

We do a simple IVF_FLAT index for our vector.

IVF_FLAT: Quantization-based index

• High-speed query
• Requires a recall rate as high as possible

See: https://medium.com/@tspann/how-good-is-quantization-in-milvus-6d224b5160b0

We iterate through a number of times (we grab ten different pages and update the URL.

for pages in range(10):
    aqpage = int(pages) + 1
    details = ""
    fulllocation = ""
    data = []

    url =  'https://api.openaq.org/v2/measurements?country=US&date_from={0}&date_to={1}&limit=1000&page={2}&offset=0&sort=desc&radius=1000&order_by=datetime'.format(
        str(YESTERDAYS_DATE), str(TODAYS_DATE), str(aqpage) )
    headers = {"accept": "application/json"}

This builds up the REST call.

response = requests.get(url, headers=headers)

openaq2 = json.loads(response.text)

 for jsonitems in openaq2['results']:
        count = count + 1
        fulllocation = 'Location {0}: {1}, {2}, {3} @ {4},{5}'.format(jsonitems.get('locationId'),
                                                               jsonitems.get('location'),jsonitems.get('city'),jsonitems.get('country'), 
                                                               jsonitems.get('coordinates')['latitude'],jsonitems.get('coordinates')['longitude'] )
        details = 'Current Air Quality Reading for {0} is {1} {2} for {3} at {4}. Is Mobile: {5} Is Analysis: {6} Entity: {7} Sensor Type: {8}'.format( 
                                                                                          jsonitems.get('parameter'), jsonitems.get('value'), 
                                                                                          jsonitems.get('unit'),
                                                                                          fulllocation, jsonitems.get('date')['local'],
                                                                                          jsonitems.get('isMobile'), jsonitems.get('isAnalysis'),
                                                                                          jsonitems.get('entity'), jsonitems.get('sensorType')
        data.append({ "locationId": int(jsonitems.get('locationId')),
                      "location": str(jsonitems.get('location','')), 
                      "parameter": str(jsonitems.get('parameter','')), 
                      "value": float(jsonitems.get('value')), 
                      "datelocal": str(jsonitems.get('date')['local']), 
                      "unit": str(jsonitems.get('unit','')), 
                      "latitude": float(jsonitems.get('coordinates')['latitude']), 
                      "longitude": float(jsonitems.get('coordinates')['longitude']), 
                      "country": str(jsonitems.get('country','')), 
                      "city": str(jsonitems.get('city','')), 
                      "isMobile": str(jsonitems.get('isMobile','')), 
                      "isAnalysis": str(jsonitems.get('isAnalysis','')), 
                      "entity": str(jsonitems.get('entity','')), 
                      "sensorType": str(jsonitems.get('sensorType','')), 
                      "vector": model(details), "details": str(details), "fulllocation": str(fulllocation)})

We iterate through the 1,000 JSON rows and parse them out. We then append to our data object to insert into Milvus.

We build up a small document with our full location from that record. We also build up a detail record that will become our vectorized data. This is our report of the Air Quality for this location. We could use libraries to make this HTML, Markdown, RTF or PDF, but it’s so small I didn’t think this was necessary. A future enhancement is to add some other data for that location that we can lookup like weather, local planes, local news, local wikipedia page, etc. This can be a detailed multipage report if need be. We then use the model(details) method to vectorize this for insert into Milvus. I have a counter for seeing how many rows we process.

We then insert into Milvus this batch of data.

res = milvus_client.insert(collection_name=“openaqmeasurements”, data=data)

We then keep iterating. More and more data. There are other libraries and methods to do this. We will explore Spark, Kafka and other libraries. Please provide suggestions as ones you would like to see. I am also looking into Java, C#, Go and batch imports.

Search and Browse Data

You connect to your Milvus cluster (or Milvus-Lite or Standalone), then you can run a search.

You can pass in a vector to search on via the model from Milvus. We then pass in a details String from our insert.

You can keep an example or build one with test data if you wish or what you may want to query on such as location, parameter, values, etc… You can use the Faker library, I have a good example here:

GitHub - tspannhw/FLaNK-Py-DataSimulator: Faker

For my search was just the last row I inserted.

details = 'Current Air Quality Reading for {0} is {1} {2} for {3} at {4}. Is Mobile: {5} Is Analysis: {6} Entity: {7} Sensor Type: {8}'.format( 
 jsonitems.get('parameter'), jsonitems.get('value'), 
 jsonitems.get('unit'),
 fulllocation, jsonitems.get('date')['local'],
 jsonitems.get('isMobile'), jsonitems.get('isAnalysis'),
 jsonitems.get('entity'), jsonitems.get('sensorType'))

from pymilvus import model
from pymilvus.model.dense import SentenceTransformerEmbeddingFunction

model = SentenceTransformerEmbeddingFunction(‘all-MiniLM-L6-v2’,device=’cpu’ )

I am using a simple Open Source Sentence Transformer Embedding Function and we only have a CPU for this one. If you have a good GPU, put that in.

For the search, I add a filter to limit to certain location’s. I also limited results to 5 and picked my output fields (“location”, “details”, “parameter”,”value”, “datelocal”,”fulllocation”).

results = milvus_client.search(
    OAQ3_COLLECTION_NAME,
    data=[model(details)],
    filter='location like "%H%"',
    output_fields=["location", "details", "parameter","value", "datelocal","fulllocation"],
    limit=5
)

for result in results:
    for hit in result[:5]:
        location = hit["entity"]["location"]
        details = hit["entity"]["details"]
        parameter = hit["entity"]["parameter"]
        value = hit["entity"]["value"]
        datelocal = hit["entity"]["datelocal"]
        fulllocation = hit["entity"]["fulllocation"]

        print(f"{location} {details} {parameter}={value} {datelocal} {fulllocation}" )

CHI_COM Current Air Quality Reading for o3 is 0.032 ppm for Location 424: CHI_COM, None, US @ 41.7547,-87.7136 at 2024-08-15T18:00:00-05:00. Is Mobile: False Is Analysis: None Entity: Governmental Organization Sensor Type: reference grade o3=0.03200000151991844 2024-08-15T18:00:00-05:00 Location 424: CHI_COM, None, US @ 41.7547,-87.7136
CHI_COM Current Air Quality Reading for o3 is 0.033 ppm for Location 424: CHI_COM, None, US @ 41.7547,-87.7136 at 2024-08-15T17:00:00-05:00. Is Mobile: False Is Analysis: None Entity: Governmental Organization Sensor Type: reference grade o3=0.032999999821186066 2024-08-15T17:00:00-05:00 Location 424: CHI_COM, None, US @ 41.7547,-87.7136
CHI_COM Current Air Quality Reading for o3 is 0.025 ppm for Location 424: CHI_COM, None, US @ 41.7547,-87.7136 at 2024-08-15T12:00:00-05:00. Is Mobile: False Is Analysis: None Entity: Governmental Organization Sensor Type: reference grade o3=0.02500000037252903 2024-08-15T12:00:00-05:00 Location 424: CHI_COM, None, US @ 41.7547,-87.7136
CHI_COM Current Air Quality Reading for o3 is 0.039 ppm for Location 424: CHI_COM, None, US @ 41.7547,-87.7136 at 2024-08-15T14:00:00-05:00. Is Mobile: False Is Analysis: None Entity: Governmental Organization Sensor Type: reference grade o3=0.039000000804662704 2024-08-15T14:00:00-05:00 Location 424: CHI_COM, None, US @ 41.7547,-87.7136
CHI_COM Current Air Quality Reading for o3 is 0.023 ppm for Location 424: CHI_COM, None, US @ 41.7547,-87.7136 at 2024-08-15T00:00:00-05:00. Is Mobile: False Is Analysis: None Entity: Governmental Organization Sensor Type: reference grade o3=0.023000000044703484 2024-08-15T00:00:00-05:00 Location 424: CHI_COM, None, US @ 41.7547,-87.7136

Our results are displayed above. Your results will vary as air quality is constantly changing and includes many thousands of locations.

RAG with Milvus, LangChain, Python and Air Quality Reports

Term: Retrieval Augmented Generation (RAG)

Let’s use our vector database of Air Quality readings to feed our LLM and get proper answers to Air Quality questions.

Zilliz

We will use LangChain, Ollama and Milvus for our RAG app.

from langchain_milvus import Milvus

embeddings = HuggingFaceEmbeddings(model_name=”all-MiniLM-L6-v2")

We build our Vector Store for LangChain usage.

vector_store = Milvus(
embedding_function=embeddings,
collection_name=“openaqmeasurements”,
primary_field = “id”,
text_field=”details”,
connection_args={“uri”: “http://MYLOCALIP:19530”},
)

def run_query() -> None:
llm = Ollama(
model=”llama3",
callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
stop=[“<|eot_id|>”],
)

query = input(“\nQuery: “)
prompt = hub.pull(“rlm/rag-prompt”, api_url=”https://api.hub.langchain.com")

qa_chain = RetrievalQA.from_chain_type(
llm, retriever=vector_store.as_retriever(collection = OAQ3_COLLECTION_NAME, expr=”parameter == ‘o3’”))

result = qa_chain({“query”: query})
print(result)

if __name__ == “__main__”:
while True:
run_query()

We then will call the llama3 model hosted locally by Ollama. If you don’t have Ollama installed, I recommend it. You can install it easily almost everywhere. We will use the LangChain hub to pull rlm/rag-prompt. We use LangChain’s RetrievalQA with Milvus filtered by parameter of o3 air quality. O3 is Ozone, something we definitely want to monitor. When air quality gets bad, remember to stay inside or wear a N95 mask.

Ollama

Example Query: What is the Currenty Air Quality Reading for pm10 in Location Mark Twain

This is a run of our small RAG application.

Query: What is the Currenty Air Quality Reading for pm10 in Location Mark Twain
Based on the provided context, I can see that there are multiple readings for pm10 at Location 255: Mark Twain SP. The readings are:

* 15.0 µg/m³ at 2024–08–15T00:00:00–05:00
* 12.0 µg/m³ at 2024–08–15T07:00:00–05:00 and 2024–08–15T08:00:00–05:00
* 16.0 µg/m³ at 2024–08–15T09:00:00–05:00

The current air quality reading for pm10 in Location Mark Twain would be the most recent reading, which is:

**12.0 µg/m³**

(Note: The question doesn’t specify a particular time frame, so I’m assuming you’re asking about the current air quality reading at any point during August 15th.){‘query’: ‘What is the Currenty Air Quality Reading for pm10 in Location Mark Twain’, ‘result’: “Based on the provided context, I can see that there are multiple readings for pm10 at Location 255: Mark Twain SP. The readings are:\n\n* 15.0 µg/m³ at 2024–08–15T00:00:00–05:00\n* 12.0 µg/m³ at 2024–08–15T07:00:00–05:00 and 2024–08–15T08:00:00–05:00\n* 16.0 µg/m³ at 2024–08–15T09:00:00–05:00\n\nThe current air quality reading for pm10 in Location Mark Twain would be the most recent reading, which is:\n\n**12.0 µg/m³**\n\n(Note: The question doesn’t specify a particular time frame, so I’m assuming you’re asking about the current air quality reading at any point during August 15th.)”}

Thanks for doing RAG with Air Quality, this can also apply to doing RAG for Manufacturing / IoT and similiar use cases for sensors, devices, machines and real-world environmental status. As vectors, AI, GPUs and more drive us to have live interactions with the real world as IoT, Sensors, Cameras, Microphones, GPS, 5G, AI, Vectors, NVIDIA, Milvus and more constantly accelerate abilities with improved hardware.

See you soon.

Unstructured Data Meetup · Events Calendar

SOURCE CODE

GitHub - tspannhw/AIM-AirQuality: Air Quality Readings

RESOURCES

AirNow API Documentation

Wildfires… Air Quality… Time to fire up the sensors… And Start FLaNKing…

GitHub - tspannhw/airquality-consumer: AirQuality Apache Pulsar Consumer

Let’s Monitor All The Things

Tracking Air Quality with Apache NiFi, Cloudera Data Science Workbench, Pyspark and Parquet

Let's Monitor The Conditions at the Conference with Timothy Spann and David Kjerrumgaard | Current…

GitHub - tspannhw/airquality: Spring Boot, JDK 11, Apache Pulsar 2.9.1, REST, AirNow

CoC23_ Let's Monitor The Conditions at the Conference

GitHub - tspannhw/flank-airquality: Spring - Kafka - FLaNK Stack Air Quality

Milvus Adventures August 14, 2024

AI and Vectors in The Sky

Challenges in Structured Document Data Extraction at Scale with LLMs

Local RAG Setup with Llama 3, Ollama, Milvus & LangChain - Zilliz blog

ollama_local_rag/rag_berlin_parliament.py at main · stephen37/ollama_local_rag

Milvus | 🦜️🔗 LangChain

langchain_milvus.vectorstores.milvus.Milvus - 🦜🔗 LangChain 0.2.13

Retrieval-Augmented Generation (RAG) with Milvus and LangChain | Milvus Documentation

Optimizing Multi-Agent Systems with Mistral Large, Nemo, and Llama-agents - Zilliz blog

Data In Motion

Harnessing the Power of NiFi: Building a Seamless Flow to Ingest PM2.5

Tracking Air Quality with Apache NiFi, Cloudera Data Science Workbench, Pyspark and Parquet

Wildfires… Air Quality… Time to fire up the sensors… And Start FLaNKing…

IoT Series: MiNiFi Agent on Raspberry Pi 4 with Enviro+ Hat For Environmental Monitoring and…

Using GrovePi with Raspberry Pi and MiNiFi Agents for Data Ingest to Parquet, Kudu, ORC, Kafka…

Powering Edge AI for Sensor Reading with RPI and Breakout Garden (EFM, NiFi, MiNiFi Agents)

Easy Deep Learning in Apache NiFi with DJL

Get Milvused!

Vector database — Milvus

Read my Newsletter every week!

AIM Weekly 17 June 2024

For more cool Unstructured Data, AI and Vector Database videos check out the Milvus vector database videos here:

Zilliz

x.com

x.com

https://www.linkedin.com/company/zilliz/

https://www.linkedin.com/in/timothyspann/

Join the Milvus Discord Server!

https://milvusio.medium.com

Open Source Vector Databases