In the age of LLMs, accidentally logging a customer’s phone number or feeding Personally Identifiable Information (PII) into a chatbot within an enterprise can be a nightmare scenario. If you are developing GenAI apps, relying solely on simple “Find & Replace” scripts is no longer sufficient.

Enter Microsoft Presidio, the open-source standard for data protection.

What is Presidio?

Presidio is an advanced SDK from Microsoft, designed to identify and anonymize sensitive data (PII) in text and images. This includes information such as credit card numbers, names, locations, social security numbers, bitcoin wallets, US phone numbers, financial data and more.

It moves beyond Regular Expressions (Regex) by using Named Entity Recognition (NER). It understands that “Washington” in “George Washington” is a Person, but in “Washington DC” it is a Location.

Presidio Detection Flow

The Architecture

Presidio is built on a Decoupled Architecture. It doesn’t just “fix” text, it separates the process into two distinct stages: Analysis and Anonymization. This allows enterprises to detect PII in one stage (audit) and redact it in another (export).

1. The Analyzer:

The AnalyzerEngine acts as the detective. It scans unstructured text and returns a list of "Findings" (coordinates, scores, and types).

• NLP-Based Detection: Utilizes models like spaCy or Stanza to understand grammar and context.
• Checksums: Validates logic (e.g., running the Luhn algorithm to ensure a Credit Card number is mathematically valid).
• Context Awareness: Increases confidence scores if it sees relevant keywords nearby.
• Example: The number 888-555-0199 alone might get a low score. But when it appears as Phone: 888-555-0199, it gets a high score because the word "Phone" is nearby.

Input text:

text = "My Name is Vinay. My phone number is 888-555-0199"


Analyzer Output:

[type: PERSON, start: 11, end: 16, score: 0.85, type: PHONE_NUMBER, start: 37, end: 49, score: 0.75]

2. The Anonymizer:

The AnonymizerEngine is the editor. It takes the Findings from the Analyzer and applies specific operations.

• Redact: Remove entirely.
• Mask: Replace with * (e.g., ***-**-1234).
• Hash: One-way encryption (useful for analytics, such as counting unique users without revealing their identities)
• Encrypt: Implement reversible encryption (AES) so that authorized personnel can restore the data later.

Actions: Mask (***-1234), Redact (<HIDDEN>), Hash, or Replace with synthetic data.

Anonymizer Output:

text: My Name is <PERSON>. My phone number is <PHONE_NUMBER>

items:
[
    {'start': 40, 'end': 54, 'entity_type': 'PHONE_NUMBER', 'text': '<PHONE_NUMBER>', 'operator': 'replace'},
    {'start': 11, 'end': 19, 'entity_type': 'PERSON', 'text': '<PERSON>', 'operator': 'replace'}
]

Implementation

Let’s build a “Hello World” pipeline.

pip install presidio-analyzer presidio-anonymizer spacy
python -m spacy download en_core_web_lg

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

# 1. Initialize Engines
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

# 2. User Text
text = "Contact support@example.com or call 212-555-0199 regarding case 998877."

# 3. Analyze (Detect)
# We only care about Phone Numbers and Emails for this pass
results = analyzer.analyze(text=text, entities=["PHONE_NUMBER", "EMAIL_ADDRESS"], language='en')

# 4. Anonymize (Redact)
# We define specific rules for specific entities
anonymized_result = anonymizer.anonymize(
    text=text,
    analyzer_results=results,
    operators={
        "PHONE_NUMBER": OperatorConfig("mask", {"type": "mask", "masking_char": "*", "chars_to_mask": 12, "from_end": True}),
        "EMAIL_ADDRESS": OperatorConfig("replace", {"new_value": "<EMAIL_REDACTED>"}),
    }
)

print(f"Original: {text}")
print(f"Cleaned:  {anonymized_result.text}")

Output:

Original: Contact support@example.com or call 212-555-0199 regarding case 998877.
Cleaned:  Contact <EMAIL_REDACTED> or call ************ regarding case 998877.

Custom Recognizers

Standard Presidio is effective for detecting generic data. But your company has specific data: Order IDs, Employee Badges, or Project Codes.

1. Custom Recognizers with Context

Let’s say your internal “Shipment IDs” look like SHIP-ABC123. A simple Regex might catch them, but it might also catch random serial numbers. We use Context to fix this.

You tell Presidio: “Only flag SHIP-ABC123 if words like 'package', 'delivery', or 'status' appear within 5 words of the match." This drastically reduces false alarms compared to standard Regex scanning.

from presidio_analyzer import PatternRecognizer, Pattern

# 1. Define the Pattern
shipment_pattern = Pattern(name="shipment_pattern", regex=r"SHIP-[A-Z]{3}\d{3}", score=0.5)

# 2. Define Context
# If these words appear nearby, we boost the score by 0.35 (Default)
context_words = ["delivery", "package", "tracking", "status"]

# 3. Create Recognizer
shipment_recognizer = PatternRecognizer(
    supported_entity="SHIPMENT_ID",
    patterns=[shipment_pattern],
    context=context_words
)

# 4. Add to Analyzer
analyzer.registry.add_recognizer(shipment_recognizer)

# Test it
text = "Please check the status of package SHIP-XYZ999."
results = analyzer.analyze(text=text, language='en')

print(f"Found: {results[0].entity_type} with score {results[0].score}")
# Result: Found SHIPMENT_ID with score 0.85 (0.5 Base + 0.35 Context Boost)

Additionally, Presidio can analyze images (such as screenshots of support tickets) and redact personally identifiable information (PII) by pixelating it.

Why It Benefits Enterprises

• Customizable: You can introduce “Custom Recognizers” to detect your specific internal data (e.g., EMPLOYEE-ID-99).
• Context Aware: It considers surrounding words (“call”, “fax”, “tracking”) to boost confidence scores, reducing false positives.
• Scalable: Deploy it as a Docker Microservice or run it as a UDF on Spark/Databricks for Big Data.
• Language Agnostic: Supports multiple languages via different NLP backends.
• Auditable: Because detection is separate from redaction, you can log what was found (e.g., “Found Credit Card with 90% confidence”) without logging the actual data.

Conclusion

Avoid creating your own PII filters; they can be brittle, risky, and difficult to maintain. Microsoft Presidio provides the framework to build a compliant and secure data platform without starting from scratch. By combining its pre-built intelligence with your custom business logic, you can ensure that your data remains accessible to developers but invisible to attackers.

If you enjoy simplifying complex AI topics into practical insights, consider following me for more articles that go beyond the usual routine.

Let’s connect and learn together!

If you treat an AI product like a traditional SaaS application, you will likely fail.

In traditional software, the relationship between inputs and outputs is predictable:, if (x) then (y). This is deterministic. You write the code, pass the unit tests, and then ship it, moving on to the next task. But in the world of Generative AI, this predictability is broken. The system is non-deterministic; You can run the same prompt twice and receive entirely different results.

I have witnessed talented engineers spend weeks optimizing a model, only to discover they were addressing a problem that users didn’t actually care about. Even worse, they may release a “perfect” demo that behaves unpredictably as soon as a real user interacts with it.

What we need is a new roadmap.

Based on in-depth analysis of agentic workflows and modern AI engineering, here is the AI Product Lifecycle, a blueprint designed to manage the chaos of non-determinism through rigorous observation and continuous evaluation.

The Architecture of the Loop

Before we dive into the steps, take a look at the architecture below. It isn’t a straightforward path; rather, it represents a dynamic system. The most crucial component is the Evaluation Loop (steps 8–11), where the product evolves based on feedback from failures.

Core Philosophy: You are not building a static feature; you are developing an engine that improves through learning from failures.

Phase 1: The Setup

Focus: Defining the Problem, Prototyping, Metrics, & Evaluations

Most AI projects fail because they address the wrong problem, rather than due to a poor model. Here’s how to approach this crucial phase:

1. Define the Problem

It’s essential to ensure that the problem is clearly defined, well-bounded, and aligned with business goals. Consider the following questions:

• Is the problem best solved by AI or traditional software?
• Who is the end user?
• What are the edge cases?
• What are the boundaries of acceptable behaviour?

Many projects fail because they attempt to use AI for problems that could be addressed with a simple if/else statement.

2. Rapid Prototyping

Forget scale. Forget latency.

• Use Notebooks or no-code tools, small datasets and off-the-shelf models, and work with small datasets.
• During this phase, engage in extensive prompting and research to identify market tools that may help address your problem (for example, voice-to-text platforms).
• Pro Tip: This phase is for learning, not performance. If you optimize for speed here, you are wasting time.
• Document the Failures: Write down every prompt that didn’t work, as this will become your first dataset.

3. Performance Metrics

You need two types of metrics:

• North Star (Output): What are you trying to optimize for? What is the business goal? (For example, “Reduce support ticket volume.”)
• Input Metrics: Metrics that can actually drive the output metric forward. (For instance, “Reduce the average time to customer support ticket resolution.”)
• Note: Your AI targets the inputs; the business will judge the output.

4. Evaluation Rules

Before building the actual application, define your test set. Create a dataset of inputs and their expected outputs.

Inputs → Expected Outputs.

Also, define unacceptable responses, such as toxicity, hallucinations, or unsafe suggestions.

Phase 2: Exposure & Reality Check

Focus: Building PoC, Instrumenting, & Observability

5. Building a PoC

This might sound unconventional, but a successful AI Proof of Concept (PoC) might just be an Excel spreadsheet.

Utilize LLM APIs from OpenAI, Google, Anthropic, and others to quickly develop your initial user-facing application.

If you can use an LLM API to fill a spreadsheet with answers that move a metric, you have a product. At this stage, you don’t need a React frontend; your priority should be to expose the system to users right away to uncover “unknown unknowns.”

6. Instrument Everything

This is where we implement best practices for Observability in LLM based systems.

1. Log Everything: Track prompts, completions, embeddings, latency, token counts, and user feedback.
2. Metadata: Include additional information such as prompt versions, user inputs, and the model versions used.
3. Connections: Ensure that the chains are properly connected and that you know the order of operations.
4. Multimodal Data: When working with multimodal data, log various types including PDFs, images, audio, and video.
5. Chaining Outputs: Remember that the output of one LLM call often becomes the input for the next.
6. User Feedback: Always attach user feedback to the traces representing the interactions users had during the runs when the feedback was provided.

7. Observability

Simply tracking data is not enough; you also need to efficiently visualize and analyze it. This is where observability platforms come into play. They assist with effective search and visualization, prompt versioning, and the addition of automated evaluation capabilities.

You need a platform (like LangSmith, Arize, or Datadog) to visualize these traces. This process isn’t just for debugging; it’s your Prompt Registry.

Phase 3: The Evolution

Focus: Evaluating Traces, Evolving, & Exposing New Versions

This phase is where the real transformation takes place.

8. Filter for Failure

Don’t look at your successes. They teach you nothing. Filter your observability data for:

1. Negative User Feedback
2. Failing Evals

This “failure pile” will serve as your roadmap for the next sprint.

9. Evolve the Topology

Now you fix the failures. But be careful complexity is a cost. The approach to evolving the system is as follows:

• Level 1: Fix the Prompt.
• Level 2: Add RAG (Retrieval Augmented Generation).
• Level 3: Integrate Agents/Tools.
• Level 4: Develop Multi-Agent Systems.

The Golden Rule: Only move up a level if the previous level absolutely cannot solve the failing eval. Avoid creating an Agent when an improved system prompt may be sufficient.

A key concept in Evaluation Driven Development is having a failing eval dataset that you would never solve for 100%. Your goal is to achieve that but by adding more and more failing samples you never get to that 100%.

10. Expose the New Version

This step emphasizes the importance of quickly releasing new versions of the application. The feedback you receive on these updates is invaluable.

Deploying new versions fast is important for few reasons:

1. It enhances user experience by addressing existing issues.
2. Some fixes may resolve multiple unknown problems simultaneously, allowing you to tackle several bugs at once

Make sure to implement strict release testing. Always have evaluation datasets prepared to ensure that the new release performs at least as well as the previous version. Integrate these checks into your CI/CD pipelines.

11. Continuous Deployment (The Feedback Loop)

Deploy quickly. If you have your evaluation dataset prepared (from Step 4), you can rely on your CI/CD pipeline. If the new model version passes the evaluations, go ahead and deploy it.

Remember:

Build → Trace, collect feedback → Evaluate → Focus on Failing Evals and Negative Feedback → Improve the application → Iterate.

Phase 4: Production

Focus: Monitoring & Alerting

12. Monitoring as a Byproduct

If you completed Phases 2 and 3 correctly, monitoring will be effortless. You’ll already be tracking everything.

If you haven’t done so yet, consider adding advanced metrics such as Time To First Token (TTFT) and inter-token latency.

Next, set up alerts.

Warning: Avoid “Alert Fatigue.” Do not alert on every hallucination. Alert on trends (e.g., “Hallucination rate spiked 5% in the last hour”).

Summary: The Evaluation Driven Development (EDD) Mindset

The biggest shift for engineers is understanding: You will never reach 100% accuracy.

In Evaluation Driven Development, your goal is to curate a dataset of “hard failures” that you are constantly trying to solve. As you solve them, you add new hard cases to your dataset.

If you respect the loop

Plan → Build → Trace → Evaluate → Repeat

you stop guessing and start engineering.

If you enjoy breaking down complex AI topics into simple, practical ideas, consider following me for more insights that go beyond the usual routine articles.

Follow me to hear more stories and useful articles on AI. Let’s connect and learn together!

It seemed simple enough. However, I soon realized that the concept of an “agent” is actually much older and broader than just LLMs, and understanding the difference made everything about AI make a lot more sense.

If you’ve had a similar confusion, let’s walk through the key points.

What an AI Agent Really Is?

Forget about LLMs for just one minute. The classic, original definition of an AI agent is incredibly simple. It’s any system that does three things:

1. Perceives its environment (sees, hears, or reads information).
2. Thinks about what to do next based on that information.
3. Acts on the environment to work towards a goal.

That’s the core loop: Perceive -> Think -> Act.

Think about the smart thermostat on your wall. It perceives the room’s temperature. It thinks based on a simple rule (“if it’s below 68 degrees…”). And it acts by turning on the heat. That thermostat is a perfect example of a basic agent.

The Car and the Engine: A Simple Analogy

So, where do powerful LLMs fit into this? Here’s the best way I’ve found to think about it:

Think of an AI agent as a car designed to get a job done (move from Point A to Point B).

The “brain” or logic of the agent is its Engine.

Over the years, we’ve had various types of engines/logics:

• Rule-Based Logic: A simple set of if-then instructions. Think of this as a trusty, mechanical V6 engine. It does its job reliably.
• Classic ML Logic: A system that learns from data, like a movie recommendation algorithm. This is like a modern, efficient electric motor.

A car with a V6 is still a car. An agent with simple rules is still an agent.

An LLM is a brand-new, super-advanced type of engine. It’s like a jet engine that allows the car to even “fly”. Similarly, LLM agents tackle tasks we never thought possible. The introduction of this new “engine” didn’t make the others obsolete; it just gave us a new, incredibly powerful option for specific jobs.

Let’s See it in Action: An Email Bot

Imagine we want an agent to sort our emails.

1. Agent without an LLM (Rule-Based): This bot operates based on a list of rules. It scans for keywords; for instance, if it sees the word “invoice,” it moves the email to a “Finance” folder. It’s efficient and inexpensive, but lacks understanding of context.

2. Agent with an LLM called “LLM Agents”(Language-Based): This bot sends the email to an LLM and asks, “What is this about?” The LLM understands that an email with the words “here is the bill” is about finance, even without the word “invoice.” It’s flexible and understands context, but it’s also slower and more expensive to run for every single email.

So, Why Do We Mix Them Up?

The reason is simple: agents powered by LLMs are doing amazing things that get all the attention. They are so capable at understanding language and reasoning through complex problems that we’ve started using the general term “agent” to refer specifically to them (LLM Agents).

But it’s important to remember: LLM-powered agents are just one type of agent. They are a very powerful type, but they are still a subset of the bigger category.

The Main Point: Use the Right Tool for the Job

This is the most useful part of understanding the difference. If you’re building an application with multiple agents working together, not every agent needs the power of an LLM.

An agent that needs to understand user intent and provide a human-readable response is an ideal candidate for an LLM Agent. However, if you don’t require a car that flies, a jet engine isn’t necessary. You can achieve your goal with simpler engines. Similarly, a simple, rule-based agent is more suitable for tasks that don’t require natural language understanding; it’s faster, cheaper, and more reliable for specific tasks (especially in a multi-agent system).

So, the next time you hear “AI Agent,” have a clearer picture. It’s a broad concept about any system that perceives, thinks, and acts. An LLM is just one of the most exciting tools available for enhancing the “thinking” aspect of an agent.

And that’s the core of it. The world of AI is moving fast, and it’s easy for terms to get blurred and terms are used interchangeably. Hopefully, this clears up the distinction between the broad concept of an “agent” and the powerful LLMs now used to build them.

But this is just my take and what I realized over time. This article is not a source but rather a question about your understanding of agents. Does this analogy work for you? How do you think about agents? I’d love to hear your thoughts in the comments.

If you enjoy breaking down complex AI topics into simple, practical ideas, consider following me for more insights that go beyond the usual routine articles.

On the evening of July 20, we took a walk and passed by the store, remembering the free ice cream offer. We weren’t sure how to get it. The poster stated we needed to scan a QR code, download an app, and register to redeem it. After registering, I found a coupon with the QR code in the app.

As I entered the store, I was impressed by the fantastic ambiance. It was bright and full of lights. Because the ice cream was free, many people had come, including students and families, making the store quite crowded. Unsure of how to get the ice cream, we asked a staff member at the billing counter for help. She explained that we could select our ice cream flavors and toppings ourselves, and at checkout, we needed to show the QR code to get $5 off our order. I thought the free ice cream was valued at $5. They were likely doing this to promote their brand, as I often see such giveaways during the summer.

The store operated on a self-service basis with over 16 flavors available from different machines. We were instructed to take a cup and sample various flavors, blending them as we liked. There were also over 40 toppings available. We noticed that no prices were displayed, which made us wonder how they would bill us, assuming that the self-serve ice creams were valued at $5.

I took a small cup of chocolate ice cream while my friend, who loves ice cream, filled his cup generously. We added many toppings, thinking we just needed to show our coupon at checkout. However, upon reaching the billing counter, I saw them weighing the ice cream on a scale and realized that the cost was based on weight. Using the coupon would deduct $5 from the total. Until that moment, we thought we were just going to receive a free cup of ice cream.

My friend smiled and asked what if our total exceeds $5. “Then they have a successful campaign.” This was my first thought that came to my mind, and I responded to my friend. My total was $5.14, so I ended up paying the extra $0.14, and my friend’s was $6.73. After the billing experience, I was genuinely impressed by their marketing strategy, especially as a business student. I started connecting the dots regarding their tactics.

They attracted customers by using National Ice Cream Day with a poster advertising a free cup of ice cream. They introduced their brand to me, motivated me to download their app and register, and brought me inside to experience their inviting ambiance. The large cups encouraged customers to take more ice cream, while the self-service element allowed us to choose our quantity without constraint. The absence of prices shifted my focus entirely to the ice cream and toppings. By the time I reached the billing counter, I was more than happy to pay for my ice cream without initially intending to buy from that store.

After reviewing the poster again, I noticed it mentioned, “Get a free cup on 07/20/25,” with a small note stating “up to $5.” I was truly impressed with how effectively they utilized the day for their marketing campaign, achieving significant success for a small store.

They were successful in running this campaign because:
1. They made customers aware of their brand.
2. They encouraged customers to register and log in.
3. They brought customers into the store.
4. They engaged customers in self-serve ice cream, leading to sales.

After enjoying the ice cream, my friend and I decided we would definitely return one day. Their strategy was successful in attracting returning customers.

By the way, the ice cream was delicious, so I was happy with the product as well!

The Business Strategy Unpacked: More Than Just a Free Scoop

This “free” ice cream offer was a strategic initiative by a local business to achieve several key goals:

1. Brand Awareness: By promoting the offer on National Ice Cream Day, they attracted attention and introduced their store to new customers.
2. Customer Data Collection: Requiring app downloads and registrations allowed them to gather valuable contact information for future marketing.
3. Increased Store Visits: The promise of free ice cream effectively drew many people into the store.
4. Engaging Customer Experience: The inviting environment and self-service model provided a positive and memorable experience, allowing customers to create their own treats.
5. Subtle Upselling: The absence of upfront pricing and an abundance of flavors encouraged customers to take more ice cream, often resulting in them exceeding the $5 limit and making small payments.
6. Future Sales and Loyalty: The delicious ice cream and overall positive experience encouraged my friend and me to want to return, turning a one-time visit into potential repeat business.

That $0.14 I paid was a small amount for a good ice cream, but it was also a clear example of a truly effective business strategy. It’s these everyday experiences that often hold the most insightful lessons about customer engagement and business growth.

Did you find this observation interesting? I love exploring the hidden strategies behind everyday business interactions. If you have any experiences like these, write in the comments.

Follow me to hear more stories and useful articles on AI about the world of business and consumer behavior. Let’s connect and learn together!

Vinay Kiran